Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Carousel stopped working concurrently with a Buildbot 2.9 update #2025

Closed
BSHoekstra opened this issue Sep 11, 2022 · 18 comments
Closed

Carousel stopped working concurrently with a Buildbot 2.9 update #2025

BSHoekstra opened this issue Sep 11, 2022 · 18 comments
Assignees

Comments

@BSHoekstra
Copy link

Any help you can provide with this recent issue would be appreciated. Thank you, Barry :)

Here are the steps I follow to reproduce the issue:

  1. Last night, I updated/upgraded to the lastest verison of Buildbot 2.9.0-pre0-7744-g3a1e19c4e and for some reason the Automatic Tool Turret (ATT) on my lathe stopped working correctly. The ATT has worked perfectly the past year including 3 to 4 weeks ago when I last used it on a project. Note I use the HAL component, Carousel, to control the ATT. Using the current version of Buildbot, Carousel works properly the 1st time the Tool Change command is issued, but then Carousel will NOT index the ATT with additional tool change commands. There are no error messages. However, graphically, Axis shows that a tool change was executed.

  2. To trouble-shoot this issue I uninstalled Buildbot 2.9 u-space and loaded Linuxcnc u-space version 2.8.3. The ATT worked perfectly with no issues. I then uninstalled Linuxcnc u-space version 2.8.3 and reloaded Buildbot version 2.9.0-pre0-7744-g3a1e19c4e. Again the ATT would only index 1 time correctly and then not accept additional tool change commands.

  3. Note when I exit from Linuxcnc and restart it, the ATT will again index 1 time correctly and then not accept additional tool change commands.

  4. To initiate a tool change manually, I use the MDI tab in Axis and type G40 T1 M6 G43 for tool 1 or G40 T5 M6 G43 for tool 5 etc.

Information about my hardware and software:

I am using this Linux distribution and version: Buster 10
I am using this kernel version: 4.19.0-20-rt-amd64
I am running ...
[X ] A binary version from linuxcnc.org (including buildbot.linuxcnc.org)
I am using this LinuxCNC version: 2.9.0-pre0-7744-g3a1e19c4e
I am using this user interface (GUI): AXIS
I am using this interface hardware vendor and chipset: parallel port, Dell 3020

#*********************************************************

#*********************************************************
The Hal coding is as follows and has not changed in the past year.

core HAL config file

loadrt [KINS]KINEMATICS

Motion Controller, gets thread periods from ini file

loadrt [EMCMOT]EMCMOT base_period_nsec=[EMCMOT]BASE_PERIOD servo_period_nsec=[EMCMOT]SERVO_PERIOD num_joints=[KINS]JOINTS
addf motion-command-handler servo-thread
addf motion-controller servo-thread

loadrt hal_parport cfg="0xe010 out"
addf parport.0.read base-thread
addf parport.0.write base-thread
addf parport.0.reset base-thread
setp parport.0.reset-time 3000

#Use Stepgen Position p for Axes XZA and Stepgen Velocity-mode v for CAROUSEL.
loadrt stepgen step_type=0,0,0,0 ctrl_type=p,p,p,v
addf stepgen.make-pulses base-thread
addf stepgen.capture-position servo-thread
addf stepgen.update-freq servo-thread

#Break-out Card Enable Control
loadrt charge_pump
addf charge-pump base-thread
net charge-pump <= charge-pump.out

Forspindle speed display

loadrt abs count=1
addf abs.0 servo-thread

#Spindle Encoder
loadrt encoder num_chan=1
addf encoder.update-counters base-thread
addf encoder.capture-position servo-thread
setp encoder.0.position-scale 60.000000
setp encoder.0.counter-mode 1

#Spindle Speed Control
loadrt pwmgen output_type=1
addf pwmgen.make-pulses base-thread
addf pwmgen.update servo-thread

Set the spindle's top speed in RPM

setp pwmgen.0.scale 1725
setp pwmgen.0.offset 0.0
setp pwmgen.0.pwm-freq 100.0
setp pwmgen.0.dither-pwm true
net spindle-cmd-rpm => pwmgen.0.value
net spindle-on <= spindle.0.on => pwmgen.0.enable
net spindle-pwm <= pwmgen.0.pwm

net spindle-position encoder.0.position => spindle.0.revs
net spindle-velocity-feedback-rps encoder.0.velocity => spindle.0.speed-in
net spindle-index-enable encoder.0.index-enable <=> spindle.0.index-enable

net spindle-phase-a encoder.0.phase-A
net spindle-index encoder.0.phase-Z
net spindle-cmd-rpm <= spindle.0.speed-out
net spindle-cmd-rpm-abs <= spindle.0.speed-out-abs
net spindle-cmd-rps <= spindle.0.speed-out-rps
net spindle-cmd-rps-abs <= spindle.0.speed-out-rps-abs
net spindle-at-speed => spindle.0.at-speed
net spindle-ccw <= spindle.0.reverse

#*********************************************************

******** ATT Control w/ Carousel************************

#*********************************************************
loadrt carousel pockets=5,1 encoding=index num_sense=2 dir=1

addf carousel.0 servo-thread

setp stepgen.3.position-scale 55.55555555556
setp stepgen.3.maxaccel 1500
setp stepgen.3.steplen 1
setp stepgen.3.stepspace 1
setp stepgen.3.dirhold 28000
setp stepgen.3.dirsetup 28000

setp carousel.0.fwd-dc -140
setp carousel.0.rev-dc 20
setp carousel.0.rev-pulse .57

net xenable => stepgen.3.enable
net carousel-step <= stepgen.3.step
net carousel-dir <= stepgen.3.dir
net carousel-step => parport.0.pin-07-out
net carousel-dir => parport.0.pin-08-out

net tool-num iocontrol.0.tool-prep-pocket => carousel.0.pocket-number
net start-change iocontrol.0.tool-change => carousel.0.enable
net change-done carousel.0.ready => iocontrol.0.tool-changed
net motor-speed carousel.0.motor-vel => stepgen.3.velocity-cmd

net index carousel.0.sense-0
net pulse carousel.0.sense-1
net index <= parport.0.pin-11-in-not
net pulse <= parport.0.pin-12-in-not

#********************************************************************
net xenable => parport.0.pin-02-out
net zstep => parport.0.pin-03-out
setp parport.0.pin-03-out-reset 0
net zdir => parport.0.pin-04-out
net xstep => parport.0.pin-05-out
setp parport.0.pin-05-out-reset 0
net xdir => parport.0.pin-06-out
net spindle-on => parport.0.pin-09-out
net max-x <= parport.0.pin-10-in-not
net max-x => joint.0.pos-lim-sw-in

net spindle-phase-a <= parport.0.pin-13-in
net spindle-pwm => parport.0.pin-14-out
net spindle-index <= parport.0.pin-15-in
net spindle-ccw => parport.0.pin-16-out
net charge-pump => parport.0.pin-17-out

#X-AXIS
setp stepgen.0.position-scale [JOINT_0]SCALE
setp stepgen.0.steplen 1
setp stepgen.0.stepspace 0
setp stepgen.0.dirhold 28000
setp stepgen.0.dirsetup 28000
setp stepgen.0.maxaccel [JOINT_0]STEPGEN_MAXACCEL
net xpos-cmd joint.0.motor-pos-cmd => stepgen.0.position-cmd
net xpos-fb stepgen.0.position-fb => joint.0.motor-pos-fb
net xstep <= stepgen.0.step
net xdir <= stepgen.0.dir
net xenable joint.0.amp-enable-out => stepgen.0.enable

#Z-AXIS
setp stepgen.1.position-scale [JOINT_1]SCALE
setp stepgen.1.steplen 1
setp stepgen.1.stepspace 0
setp stepgen.1.dirhold 28000
setp stepgen.1.dirsetup 28000
setp stepgen.1.maxaccel [JOINT_1]STEPGEN_MAXACCEL
net zpos-cmd joint.1.motor-pos-cmd => stepgen.1.position-cmd
net zpos-fb stepgen.1.position-fb => joint.1.motor-pos-fb
net zstep <= stepgen.1.step
net zdir <= stepgen.1.dir
net zenable joint.1.amp-enable-out => stepgen.1.enable

estop loopback

net estop-out charge-pump.enable iocontrol.0.user-enable-out
net estop-out <= iocontrol.0.user-enable-out
net estop-out => iocontrol.0.emc-enable-in

#*************************************************************************

COMMENT OUT TO USE CAROUSEL

net tool-change iocontrol.0.tool-change => hal_manualtoolchange.change

net tool-changed iocontrol.0.tool-changed <= hal_manualtoolchange.changed

#*************************************************************************
loadusr -W hal_manualtoolchange
net tool-number iocontrol.0.tool-prep-number => hal_manualtoolchange.number
net tool-prepare-loopback iocontrol.0.tool-prepare => iocontrol.0.tool-prepared

@andypugh andypugh self-assigned this Sep 11, 2022
@petterreinholdtsen
Copy link
Collaborator

petterreinholdtsen commented Sep 12, 2022 via email

@andypugh
Copy link
Collaborator

I will try to look at this tomorrow.
In the meantime, you can get the version of carousel from 2.8 and compile/install with halcompile.
Ignore the recent push, that was unrelated outstanding work.

@BSHoekstra
Copy link
Author

Thank you guys for looking into this. Here are 3 additional details for clarity that I thought of today that may be helpful,

  1. I have been running Buildbot 2.9 for the past year and I update/upgrade it about every 2 - 3 weeks. I only loaded Linuxcnc u-space version 2.8.3 temporarily yesterday to trouble-shoot because that's what I knew how to do easily, and the ATT worked perfectly with 2.8.3, so I think that probably rules out hardware issues.

  2. Tonight, I realized that I had an earlier version of Buildbot 2.9.0-pre0-7123-g432oec2ea loaded on another CNC computer in my shop and I'm sure the ATT/Carousel worked okay in that version, so I hooked it up to my CNC lathe. Again, the ATT would only index 1 time correctly and then not accept additional tool change commands. So now, I'm thinking this issue may be unrelated to a Buildbot version upgrade...not sure though.

  3. All other commands, and Axis moves, etc. work perfectly.

Please note that I'm not exactly sure how to get the version of carousel from 2.8 and compile/install with halcompile but I'll do some reading and figure it out. I'll let you know how that goes.

Thank you,

Barry :)

@BSHoekstra
Copy link
Author

Hello Andy. Per your suggestion, I renamed the carousel.so file in 2.9 to carousel29.so and then copied the carousel.so file from 2.8 to 2.9. Guess what!! The ATT works perfectly again in 2.9.0-pre0-7744-g3a1e19c4e. So the bug appears to be in the lastest 2.9 carousel.so file maybe.? I hope this helps with trouble-shooting.

Thank you,

Barry :)

@andypugh
Copy link
Collaborator

andypugh commented Oct 2, 2022

I have tried to get back to this, but I don't really know what is going on, as the none of the new code should be active unless actually turned on.
Are you in a position to try compiling/installing different versions and seeing what is going on?
First though, could you go back to the buildbot version, and halscope the carousel pins during the first and second tool change. Specifically pocket-number, current-position and state. Maybe also ready and motor-vel

@BSHoekstra
Copy link
Author

Yup. No problem. I'll help troubleshoot.

I'm not an expert on software, just an Engineer, but I understand that this is a weird case, because I would think that the basic fundamental parameters/structure that makes Carousel work would have stayed the same through any potential software upgrades but yet that's where the problem seems to be. I also noted that based on Peter's note in this chain that he was using encoding=binary which seems to work okay for him, and I am using encoding=index which seems to have the issue....not sure what this points to as an issue, but it may help narrow it down.

So,

  1. Halscoping is no problem. I'll find some time later this week in the evening to get this information for you.

  2. However, can you please direct me to some written directions on how to compile & install different versions. I know that I can figure it out, because I've done this before in other software platforms, but just not Linux that I remember. Also, is it different versions of the carousel.so file or different versions of LinuxCNC that you want tested. Also, where do I find the different versions that you would like tested.

Thank you,

Barry :)

@andypugh
Copy link
Collaborator

Because "carousel" is a HAL component written in the .comp preprocessing language, it is just:

sudo halcompile --install carousel.comp

@BSHoekstra
Copy link
Author

BSHoekstra commented Oct 12, 2022

Hello Andy. Here is the Hal data you requested. Please note that I did notice a difference in Hal data between the non-working Carousl.so file and a working Carousel.so file. This is highlighted in the attached file. Please note my curser location shown next to the tool change buttons that highlight to show my sequence of steps. I hope this helps pinpoint the issue. Please let me know if you need any additional data.

Thank you,

Barry :)
Carousel Hal Data Rev0.pdf

@andypugh
Copy link
Collaborator

andypugh commented Oct 12, 2022

I really need the carousel.0.state parameter, and it would be easier to see what is happening with a halcope trace, like this:

Screenshot 2022-10-12 at 11 26 26

Here is a Halscope config file that will (hopefully) save you the trouble of setting it up. Though it might still take a bit of trouble to get used to the (weird) controls, triggering etc.
If you can then save the data for first and second tool changes to .log files (from Halscope)

Carousel.halscope.txt

(You might have to take the .txt off of the end of the filename to get Halscope to open it)

@BSHoekstra
Copy link
Author

No problem. I'll get you the information in the next 2 days.

Thank you,

Barry :)

@BSHoekstra
Copy link
Author

Workinghalscope.csv
Non-Workinghalscope.csv

Per your request here are the halscope log files using your halscope configuration file. One file shows the traces from the non-working Carousel.so file and the other shows the traces from the working Carousel.so file. BTW thank you for preconfiguring the halscope settings for me. Please let me kwow if any additional inforamation is needed.

Thank you,

Barry :)

@BSHoekstra
Copy link
Author

After looking at the 2 log files attached in my previous note, I'm thinking this is not what you were looking for. They seem to be truncated. I'm working on getting you another file with a screen shot of halscope

@BSHoekstra
Copy link
Author

BSHoekstra commented Oct 13, 2022

Okay, I think this might be the data you were expecting. Sorry I had to learn how to use HalScope tonight.
Workinghalscope file 2.csv
Non-Workinghalscope file 2.csv

WORKING

working halscope file 2

NON WORKING: Note: I forgot to lower the gain setting on the Hal Scope for motor velocity for this screen shot, but all other gain settings should be the same.

nonworking halscope file 2

For each file, the initial tool was #1. Then I selected Tool #2, After the ATT advanced to Tool #2, I then selected Tool #3. On the working Carousel.so file the tool advanced rom #1 to #2 to #3. On the non-working Carousel.so file tool advanced from #1 to #2, but would not advance again to Tool #3.

Please let me know if any additional data is needed.

Thank you,

Barry :)

@andypugh
Copy link
Collaborator

The .csv files had what I needed. It is clear that the component is getting stuck on state 7, which it never should do without an align-dc configured.
The error seems to be a missing "break;" in the switch statement, so that execution is falling through.
(schoolboy error!)

Please try the attached file. (remove the .txt from the filename, I had to add that for it to be allowed as an attachment)

carousel.comp.txt

@BSHoekstra
Copy link
Author

BSHoekstra commented Oct 14, 2022

Success!!! Fixed!! Yay! Thank you very much!! :) Will you be able to upload this new carousel.so file to the 2.9 buildbot master branch?

Just as FYI....one thing I noticed was that Halcompile is not automatically downloaded with Buildbot 2.9, I had to go back to 2.8 to compile the carousel.comp file and then transfer the .so file to 2.9. I noticed that you commented on this issue on 11 Jan 2021. I guess this is still being looked at.

One last quick thing....do know if there are any updates on the G71.X Path Issue (1146#) posted on May 3, 2021 (#1146 (comment))? I've thought more about this issue and I think that if G71.2 doesn't "recognize" any rasied edges / pockets given the part shape and direction of cut, then linuxcnc goes crazy. Given my limited knowledge of programing I would think that a few lines of code could be added as a 1st step to check for 0 raised edges / pockets and if none are found then have G71.2 generate a comment message stating that zero pockets were found for cutting. May conceptual thought is shown below graphically.

image

Thank you again Andy. It is always a pleasure and honor to work with you. I am envious of your skills and talent.

Have a nice day,

Barry :)

andypugh added a commit that referenced this issue Oct 14, 2022
Control was falling-through the switch statement to a state that should
only be active on certain configurations.


Signed-off-by: andypugh <andy@bodgesoc.org>
@andypugh
Copy link
Collaborator

I was just waiting for confirmation that the change fixed the problem. Thanks for doing the testing, it made it much easier to see where the problem was.

As for halcompile, I think that it should be installed if you install the linuxcnc-dev package?

@andypugh
Copy link
Collaborator

andypugh commented Jul 7, 2024

Lookng at this at the LinuxCNC Norway meetup, it looks like the simulated config in sim-axis-vismach-vmc_toolchange-index is also broken, so this should be something that I have the means of fixing.
Can you check that you get stuck with enable false, and ready not set, and with the position not matching the pocket number?

@BSHoekstra
Copy link
Author

In response to: Can you check that you get stuck with enable false, and ready not set, and with the position not matching the pocket number?

I just loaded: 2.10.0-pre0-3309-gfd6a133f8b

With enable false, and ready not set, and with the position NOT matching the pocket number, the auto-tool changer indexes 1 time to the correct tool and then it will not respond to the next tool change command; however, the Axis graphical display changes to show the correct tool shape/orientation/etc. every time a new tool change command is issued.

With enable false, and ready not set, and with the position matching the pocket number the auto-tool changer indexes 1 time to the correct tool and then it will not respond to the next tool change command; however, the Axis graphical display changes to show the correct tool shape/orientation/etc. every time a new tool change command is issued.

Thank you,

Barry :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants