[squeak-dev] Refactoring Browser error

[Alistair Grant]

Hi All,

If I attempt to execute the following:

| rules |

rules := RBParseTreeRewriter new.

rules replace: 'state at: `@index put: `@expr' with: 'state at:
`@index put: `@expr asC_unsigned_int'.

(rules executeTree: (MD5Plugin parseTreeFor:
#md5ProcessBuffer:withState:)) ifTrue:
[rules tree newSource inspect].
self error: 'parse tree rewrite failed'


I get "Error: End of statement list encountered" (stack trace below).
The same works and produces the expected output in Pharo 7.

It looks like the Refactoring Browser has diverged quite a bit between
Squeak and Pharo.

Who looks after RB in Squeak, and is there a plan to keep in sync with
Pharo, or the other way around, or should this be fixed independently?

Thanks,
Alistair




4 November 2018 8:01:29.599623 pm

VM: unix - Smalltalk
Image: Squeak5.3alpha [latest update: #18261]

SecurityManager state:
Restricted: false
FileAccess: true
SocketAccess: true
Working Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03
Trusted Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03/secure
Untrusted Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03/My Squeak

RBPatternParser(Object)>>error:
Receiver: a RBPatternParser
Arguments and temporary variables:
aString: 'End of statement list encounted'
Receiver's instance variables:
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
source: 'state at: `@index put: `@expr asC_unsigned_int'
comments: an OrderedCollection()
pragmas: nil

RBPatternParser(RBParser)>>parserError:
Receiver: a RBPatternParser
Arguments and temporary variables:
aString: 'End of statement list encounted'
Receiver's instance variables:
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
source: 'state at: `@index put: `@expr asC_unsigned_int'
comments: an OrderedCollection()
pragmas: nil

RBPatternParser(RBParser)>>parseStatementList:into:
Receiver: a RBPatternParser
Arguments and temporary variables:
pragmaBoolean: false
sequenceNode: RBSequenceNode()
statements: an OrderedCollection(RBMessageNode(state at: `@index put:
`@expr as...etc...
return: true
periods: an OrderedCollection()
returnPosition: nil
node: RBMessageNode(state at: `@index put: `@expr asC)
Receiver's instance variables:
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
source: 'state at: `@index put: `@expr asC_unsigned_int'
comments: an OrderedCollection()
pragmas: nil

RBPatternParser(RBParser)>>parseStatements:
Receiver: a RBPatternParser
Arguments and temporary variables:
pragmaBoolean: false
args: #()
leftBar: nil
rightBar: nil
Receiver's instance variables:
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
source: 'state at: `@index put: `@expr asC_unsigned_int'
comments: an OrderedCollection()
pragmas: nil

RBPatternParser(RBParser)>>parseExpression:
Receiver: a RBPatternParser
Arguments and temporary variables:
aString: 'state at: `@index put: `@expr asC_unsigned_int'
node: nil
Receiver's instance variables:
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
source: 'state at: `@index put: `@expr asC_unsigned_int'
comments: an OrderedCollection()
pragmas: nil

RBPatternParser class(RBParser class)>>parseExpression:onError:
Receiver: RBPatternParser
Arguments and temporary variables:
aString: 'state at: `@index put: `@expr asC_unsigned_int'
aBlock: nil
node: nil
parser: a RBPatternParser
Receiver's instance variables:
superclass: RBParser
methodDict: a MethodDictionary(#messageNodeClass->(RBPatternParser>>#messageNod...etc...
format: 65543
instanceVariables: nil
organization: ('private-classes' messageNodeClass methodNodeClass
pragmaNodeClass...etc...
subclasses: nil
name: #RBPatternParser
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Parser'

RBParser class>>parseRewriteExpression:onError:
Receiver: RBParser
Arguments and temporary variables:
aString: 'state at: `@index put: `@expr asC_unsigned_int'
aBlock: nil
Receiver's instance variables:
superclass: Object
methodDict: a MethodDictionary(#addCommentsTo:->(RBParser>>#addCommentsTo:
"a CompiledMethod...etc...
format: 65543
instanceVariables: #('scanner' 'currentToken' 'nextToken' 'errorBlock'
'source'...etc...
organization: ('private' addCommentsTo: nextToken
patchLiteralArrayToken patchLiteralMessage...etc...
subclasses: {RBPatternParser}
name: #RBParser
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Parser'

RBParser class>>parseRewriteExpression:
Receiver: RBParser
Arguments and temporary variables:
aString: 'state at: `@index put: `@expr asC_unsigned_int'
Receiver's instance variables:
superclass: Object
methodDict: a MethodDictionary(#addCommentsTo:->(RBParser>>#addCommentsTo:
"a CompiledMethod...etc...
format: 65543
instanceVariables: #('scanner' 'currentToken' 'nextToken' 'errorBlock'
'source'...etc...
organization: ('private' addCommentsTo: nextToken
patchLiteralArrayToken patchLiteralMessage...etc...
subclasses: {RBPatternParser}
name: #RBParser
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Parser'

RBStringReplaceRule>>replaceString:
Receiver: a RBStringReplaceRule
Arguments and temporary variables:
replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
Receiver's instance variables:
searchTree: RBMessageNode(state at: `@index put: `@expr)
owner: nil
verificationBlock: [closure] in RBStringReplaceRule(RBReplaceRule)>>initialize
replaceTree: nil

RBStringReplaceRule>>searchFor:replaceWith:
Receiver: a RBStringReplaceRule
Arguments and temporary variables:
searchString: 'state at: `@index put: `@expr'
replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
Receiver's instance variables:
searchTree: RBMessageNode(state at: `@index put: `@expr)
owner: nil
verificationBlock: [closure] in RBStringReplaceRule(RBReplaceRule)>>initialize
replaceTree: nil

RBStringReplaceRule class>>searchFor:replaceWith:
Receiver: RBStringReplaceRule
Arguments and temporary variables:
searchString: 'state at: `@index put: `@expr'
replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
Receiver's instance variables:
superclass: RBReplaceRule
methodDict: a MethodDictionary(#foundMatchFor:->(RBStringReplaceRule>>#foundMat...etc...
format: 65540
instanceVariables: #('replaceTree')
organization: ('matching' foundMatchFor:)
('initialize-release' methodReplaceString:...etc...
subclasses: nil
name: #RBStringReplaceRule
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Matching'

RBParseTreeRewriter>>replace:with:
Receiver: a RBParseTreeRewriter
Arguments and temporary variables:
searchString: 'state at: `@index put: `@expr'
replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
Receiver's instance variables:
searches: an OrderedCollection()
answer: nil
argumentSearches: an OrderedCollection()
context: a RBSmallDictionary
messages: nil
tree: nil

UndefinedObject>>DoIt
Receiver: nil
Arguments and temporary variables:
rules: a RBParseTreeRewriter
Receiver's instance variables:
nil


--- The full stack ---
RBPatternParser(Object)>>error:
RBPatternParser(RBParser)>>parserError:
RBPatternParser(RBParser)>>parseStatementList:into:
RBPatternParser(RBParser)>>parseStatements:
RBPatternParser(RBParser)>>parseExpression:
RBPatternParser class(RBParser class)>>parseExpression:onError:
RBParser class>>parseRewriteExpression:onError:
RBParser class>>parseRewriteExpression:
RBStringReplaceRule>>replaceString:
RBStringReplaceRule>>searchFor:replaceWith:
RBStringReplaceRule class>>searchFor:replaceWith:
RBParseTreeRewriter>>replace:

[Levente Uzonyi]

Post by Alistair Grant
Hi All,
| rules |
rules := RBParseTreeRewriter new.

I presume the version you use doesn't support underscores in selectors.
Does it work if you use a selector with no underscores in it?
How did you load RB? Did you use the Metacello configuration? (it may
not be up-to-date)

Levente

Post by Alistair Grant
[rules tree newSource inspect].
self error: 'parse tree rewrite failed'
I get "Error: End of statement list encountered" (stack trace below).
The same works and produces the expected output in Pharo 7.
It looks like the Refactoring Browser has diverged quite a bit between
Squeak and Pharo.
Who looks after RB in Squeak, and is there a plan to keep in sync with
Pharo, or the other way around, or should this be fixed independently?
Thanks,
Alistair
4 November 2018 8:01:29.599623 pm
VM: unix - Smalltalk
Image: Squeak5.3alpha [latest update: #18261]
Restricted: false
FileAccess: true
SocketAccess: true
Working Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03
Trusted Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03/secure
Untrusted Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03/My Squeak
Receiver: a RBPatternParser
aString: 'End of statement list encounted'
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
comments: an OrderedCollection()
pragmas: nil
Receiver: a RBPatternParser
aString: 'End of statement list encounted'
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
comments: an OrderedCollection()
pragmas: nil
Receiver: a RBPatternParser
pragmaBoolean: false
sequenceNode: RBSequenceNode()
return: true
periods: an OrderedCollection()
returnPosition: nil
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
comments: an OrderedCollection()
pragmas: nil
Receiver: a RBPatternParser
pragmaBoolean: false
args: #()
leftBar: nil
rightBar: nil
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
comments: an OrderedCollection()
pragmas: nil
Receiver: a RBPatternParser
node: nil
scanner: a RBPatternScanner
currentToken: RBShortAssignmentToken
nextToken: nil
errorBlock: nil
comments: an OrderedCollection()
pragmas: nil
Receiver: RBPatternParser
aBlock: nil
node: nil
parser: a RBPatternParser
superclass: RBParser
methodDict: a MethodDictionary(#messageNodeClass->(RBPatternParser>>#messageNod...etc...
format: 65543
instanceVariables: nil
organization: ('private-classes' messageNodeClass methodNodeClass
pragmaNodeClass...etc...
subclasses: nil
name: #RBPatternParser
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Parser'
Receiver: RBParser
aBlock: nil
superclass: Object
"a CompiledMethod...etc...
format: 65543
instanceVariables: #('scanner' 'currentToken' 'nextToken' 'errorBlock'
'source'...etc...
organization: ('private' addCommentsTo: nextToken
patchLiteralArrayToken patchLiteralMessage...etc...
subclasses: {RBPatternParser}
name: #RBParser
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Parser'
Receiver: RBParser
superclass: Object
"a CompiledMethod...etc...
format: 65543
instanceVariables: #('scanner' 'currentToken' 'nextToken' 'errorBlock'
'source'...etc...
organization: ('private' addCommentsTo: nextToken
patchLiteralArrayToken patchLiteralMessage...etc...
subclasses: {RBPatternParser}
name: #RBParser
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Parser'
Receiver: a RBStringReplaceRule
owner: nil
verificationBlock: [closure] in RBStringReplaceRule(RBReplaceRule)>>initialize
replaceTree: nil
Receiver: a RBStringReplaceRule
owner: nil
verificationBlock: [closure] in RBStringReplaceRule(RBReplaceRule)>>initialize
replaceTree: nil
Receiver: RBStringReplaceRule
superclass: RBReplaceRule
methodDict: a MethodDictionary(#foundMatchFor:->(RBStringReplaceRule>>#foundMat...etc...
format: 65540
instanceVariables: #('replaceTree')
organization: ('matching' foundMatchFor:)
('initialize-release' methodReplaceString:...etc...
subclasses: nil
name: #RBStringReplaceRule
classPool: nil
sharedPools: nil
environment: Smalltalk
category: #'AST-Core-Matching'
Receiver: a RBParseTreeRewriter
searches: an OrderedCollection()
answer: nil
argumentSearches: an OrderedCollection()
context: a RBSmallDictionary
messages: nil
tree: nil
UndefinedObject>>DoIt
Receiver: nil
rules: a RBParseTreeRewriter
nil
--- The full stack ---
RBParse

[Alistair Grant]

Hi Levente,

Post by Levente Uzonyi

Post by Alistair Grant
Hi All,
| rules |
rules := RBParseTreeRewriter new.

I presume the version you use doesn't support underscores in selectors.
Does it work if you use a selector with no underscores in it?

That's it, thanks! (it works without the underscores)

I hate it when I forget to include version information, sorry:

Squeak5.3alpha
latest update: #18261
Image format 68021 (64 bit)

AST-Core-eem.97
Refactoring-Changes-eem.22
Refactoring-Core-ul.162
Refactoring-Environment-eem.11
Refactoring-Squeak-Platform-cwp.2
Refactoring-Tests-Changes-lr.16
Refactoring-Tests-Core-lr.56
Refactoring-Tests-Environment-eem.5

Post by Levente Uzonyi
How did you load RB? Did you use the Metacello configuration? (it may
not be up-to-date)

All the above appear to be the latest versions on
http://www.squeaksource.com/rb.

This is a vmmaker image, they were all loaded through:

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/BuildSqueakSpurTrunkVMMakerImage.st

It sounds like there may be an updated version of RB that already
addresses this?

Thanks again,
Alistair

Post by Levente Uzonyi
Levente

Post by Alistair Grant
[rules tree newSource inspect].
self error: 'parse tree rewrite failed'
I get "Error: End of statement list encountered" (stack trace below).
The same works and produces the expected output in Pharo 7.
It looks like the Refactoring Browser has diverged quite a bit between
Squeak and Pharo.
Who looks after RB in Squeak, and is there a plan to keep in sync with
Pharo, or the other way around, or should this be

[Levente Uzonyi]

Hi Alistair,

I have found the cause of the problem, and it's easy to work the issue
around. Just evaluate the following in a workspace:

Scanner allowUnderscoreAsAssignment: false.
RBScanner initializeClassificationTable

The cause of the error is that RBScanner considers _ to be #special
instead of #alphabetic when Scanner class >> #allowUnderscoreAsAssignment
returns true.
In Pharo this is not an issue, because underscore assignments have been
abandoned and underscores in selectors are allowed permanently.
But in Squeak everything can happen (so underscore can either be #special
or #alphabetic depending on its context) and RBScanner would need some
rewrite to support those cases.

I suggest the line

Scanner allowUnderscoreAsAssignment: false

be added to BuildSqueakSpurTrunkVMMakerImage.st before RB is loaded to
avoid this issue in future images (because VMMaker code shouldn't rely
on underscore assignments being supported).

Levente

Post by Alistair Grant
Hi Levente,

Post by Levente Uzonyi

Post by Alistair Grant
Hi All,
| rules |
rules := RBParseTreeRewriter new.

I presume the version you use doesn't support underscores in selectors.
Does it work if you use a selector with no underscores in it?

That's it, thanks! (it works without the underscores)
Squeak5.3alpha
latest update: #18261
Image format 68021 (64 bit)
AST-Core-eem.97
Refactoring-Changes-eem.22
Refactoring-Core-ul.162
Refactoring-Environment-eem.11
Refactoring-Squeak-Platform-cwp.2
Refactoring-Tests-Changes-lr.16
Refactoring-Tests-Core-lr.56
Refactoring-Tests-Environment-eem.5

Post by Levente Uzonyi
How did you load RB? Did you use the Metacello configuration? (it may
not be up-to-date)

All the above appear to be the latest versions on
http://www.squeaksource.com/rb.
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/BuildSqueakSpurTrunkVMMakerImage.st
It sounds like there may be an updated version of RB that already
addresses this?
Thanks again,
Alistair

Post by Levente Uzonyi
Levente

Post by Alistair Grant
[rules tree newSource inspect].
self error: 'parse tree rewrite failed'
I get "Error: End of statement list encountered" (stack trace below).
The same works and produces the expected output in Pharo 7.
It looks like the Refactoring Browser has diverged quite a bit between
Squeak and Pharo.
Who looks after RB in Squeak, and is there a plan to keep in sync with
Pharo, or the other way around, or should this be fixed independently?
Than

[Alistair Grant]

Hi Levente,

Post by Levente Uzonyi
Hi Alistair,
I have found the cause of the problem, and it's easy to work the issue
Scanner allowUnderscoreAsAssignment: false.
RBScanner initializeClassificationTable

Yep, that fixes it. Thanks very much for tracking this down.

Post by Levente Uzonyi
The cause of the error is that RBScanner considers _ to be #special
instead of #alphabetic when Scanner class >> #allowUnderscoreAsAssignment
returns true.
In Pharo this is not an issue, because underscore assignments have been
abandoned and underscores in selectors are allowed permanently.
But in Squeak everything can happen (so underscore can either be #special
or #alphabetic depending on its context) and RBScanner would need some
rewrite to support those cases.
I suggest the line
Scanner allowUnderscoreAsAssignment: false
be added to BuildSqueakSpurTrunkVMMakerImage.st before RB is loaded to
avoid this issue in future images (because VMMaker code shouldn't rely
on underscore assignments being supported).

Shall do.

Thanks again,
Alistair

Post by Levente Uzonyi
Levente

Post by Alistair Grant
Hi Levente,

Post by Levente Uzonyi

Post by Alistair Grant
Hi All,
| rules |
rules := RBParseTreeRewriter new.

I presume the version you use doesn't support underscores in selectors.
Does it work if you use a selector with no underscores in it?

That's it, thanks! (it works without the underscores)
Squeak5.3alpha
latest update: #18261
Image format 68021 (64 bit)
AST-Core-eem.97
Refactoring-Changes-eem.22
Refactoring-Core-ul.162
Refactoring-Environment-eem.11
Refactoring-Squeak-Platform-cwp.2
Refactoring-Tests-Changes-lr.16
Refactoring-Tests-Core-lr.56
Refactoring-Tests-Environment-eem.5

Post by Levente Uzonyi
How did you load RB? Did you use the Metacello configuration? (it may
not be up-to-date)

All the above appear to be the latest versions on
http://www.squeaksource.com/rb.
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/BuildSqueakSpurTrunkVMMakerImage.st
It sounds like there may be an updated version of RB that already
addresses this?
Thanks again,
Alistair

Post by Levente Uzonyi
Levente

Post by Alistair Grant
[rules tree newSource inspect].
self error: 'parse tree rewrite failed'
I get "Error: End of statement list encountered" (stack trace below).
The same works and produces the expected output in Pharo 7.
It looks like the Refactoring Browser has diverged quite a bit between
Squeak and Pharo.
Who looks after RB in Squeak, and is there a plan to keep in sync with
Pharo, or the other way around, or should this be fixed independently?
Thanks,