NoVA TestBeam Electronic Logbook

Logbook entries

(sorted by Created ) Sort by Updated

Category: subcategories	Words:
Quick search:

<newer

older>

collapse all expand all auto-refresh

Legend for symbols:

Private entry. ! Entry has newer comment.

Threaded entry.

Sticky entry.

02/25/2021

Detector DAQ [lackey32] Expert TDU Updating expected offset time for 1Hz in TDU status code. For some reason since sometime yesterday the TDUs have been showing a larger offset, around 155 clock ticks. Abhilash and David tried scrubbing the TDUs, but this didn't fix the issue. Something with the timing of the 1 Hz ($8F) signal must have changed. We saw it change once before, coming out of the 2019 shutdown. Interestingly, it appears to have changed back to the offset we had in period 1 (spring 2019). After some digging, I've found the correct code to update. • Added SHM_Utilities to testRel_TB17_00_00 and edited SHM_Utilities/cxx/src/DumpSpillHistory.cc. • Reverted offset from 1115 to 960. • Built package. TDU status monitor looked better instantly, didn't even need to restart anything.

9004 02/25/2021 18:42:53

Detector DAQ Expert TDU

Teresa Lackey

Entry Subject: Updating expected offset time for 1Hz in TDU status code.

For some reason since sometime yesterday the TDUs have been showing a larger offset, around 155 clock ticks. Abhilash and David tried scrubbing the TDUs, but this didn't fix the issue. Something with the timing of the 1 Hz ($8F) signal must have changed. We saw it change once before, coming out of the 2019 shutdown. Interestingly, it appears to have changed back to the offset we had in period 1 (spring 2019). After some digging, I've found the correct code to update.

• Added SHM_Utilities to testRel_TB17_00_00 and edited SHM_Utilities/cxx/src/DumpSpillHistory.cc.
• Reverted offset from 1115 to 960.
• Built package.

TDU status monitor looked better instantly, didn't even need to restart anything.

02/24/2021

Operations [torun] Magnet

Controlled Access M1 swept

9003 02/24/2021 16:43:24

Operations Magnet

Yagmur Torun

Entry Subject: M1 swept

Form:	Controlled Access
Controlled Access Leader:	Yagmur
Others on Controlled Access:	Mike Wallbank
Run Coordinator Approval:	True
Shifter Informed:	False
Controlled Access Purpose:	Magnet area sweep before turning on the M1 PS.
Comments:

02/11/2021

Detector DAQ [lackey32] Expert

Computing Checklist

9002 02/11/2021 09:34:02

Detector DAQ Expert

Teresa Lackey

Entry Subject:

Form:	Computing Checklist
:
Is disk space on master okay?:	Yes
Is disk space on the datadisk okay?:	Yes
Are FTS transfers normal?:	Yes
:
Is the page up-to-date?:	Yes
Are there any missing raw files?:	No
Are there any missing database entries?:	Missing DDActivity1 files Missing Beamline files Missing TBSpill files
Are there any missing root files?:	No
Comments::	4 runs missing from the database. I'll deal with them on Sunday as usual.

02/08/2021

Detector DAQ [lackey32] Expert

Computing Checklist

9001 02/08/2021 10:34:43

Detector DAQ Expert

Teresa Lackey

Entry Subject:

Form:	Computing Checklist
:
Is disk space on master okay?:	Yes
Is disk space on the datadisk okay?:	Yes
Are FTS transfers normal?:	Yes
:
Is the page up-to-date?:	Yes
Are there any missing raw files?:	No
Are there any missing database entries?:	No
Are there any missing root files?:	No
Comments::

02/05/2021

Operations [torun] Expert TOF Trigger

Controlled Access

9000 02/05/2021 18:22:33

Operations Expert TOF Trigger

Yagmur Torun

Entry Subject:

Form:	Controlled Access
Controlled Access Leader:	Mike Wallbank
Others on Controlled Access:	Yagmur
Run Coordinator Approval:	True
Shifter Informed:	True
Controlled Access Purpose:	DAQ timing calibration
Comments:	This was earlier in the day (about 1:30-4:30pm). Collected front-end/trigger timing data for TOFs, paddles and trigger board, data in Beamline runs 3697-3707, see NOvA-main-ECL entry 184107 for details.

02/05/2021

Detector DAQ [lackey32] Expert Nearline OnMon/EVD Setting up test-db-02 Since test-db is in a rack by itself that SCD would like to shutoff, Seth setup a new machine for us: novadaq-test-db-02. I'm starting to get the machine setup now. Once everything is setup I'll move EVD, OnMon, and all Nearline processing over. I'm not sure how long everything will take, so we might not get everything moved over today.

8999 02/05/2021 14:26:36

Detector DAQ Expert Nearline OnMon/EVD

Teresa Lackey

Entry Subject: Setting up test-db-02

Since test-db is in a rack by itself that SCD would like to shutoff, Seth setup a new machine for us: novadaq-test-db-02. I'm starting to get the machine setup now. Once everything is setup I'll move EVD, OnMon, and all Nearline processing over. I'm not sure how long everything will take, so we might not get everything moved over today.

Comments:

Teresa Lackey (lackey32) 02/05/2021 15:47:29: I rsynced everything from /data/nusoft and /data/nearline-data over to test-db-02. I’m going to edit the scripts in /home/novacr02/DAQ-Desktop-Utilities and try starting EVD and OnMon on the new machine.
Teresa Lackey (lackey32) 02/05/2021 16:17:26: Installed openssl-devel and libglvnd-opengl. EVD won’t open (on either machine) but the error on test-db-02 matches that on test-db, so I’m fairly confident things will work. The TOF calibration work that’s going on right now is really screwing up the detector data, so I’m going to pause this work for now.
Teresa Lackey (lackey32) 02/05/2021 17:09:15: Restarted EVD and OnMon on test-db-02 and edited CheckDSORes script to open on 02 as well.
Teresa Lackey (lackey32) 02/05/2021 17:31:07: /nusoft/app is mounted as a read-only filesystem, so I can’t start up the nearline jobs on this machine yet. I responded to the ticket, and hopefully once that’s in place switching over will be as simple as installing the crontab on test-db-02 and removing the current one on test-db.
Teresa Lackey (lackey32) 02/09/2021 15:43:15: /nusoft/app should now be writeable. I just removed the crontab from test-db and installed it on test-db-02. Did a final rsync of nearline-data so the nearline wouldn’t be stuck reprocessing older files. I’ll keep an eye on the Nearline and ask Gavin to let me know if anything for TB falls behind.

02/03/2021

Operations [torun]

Controlled Access

8998 02/03/2021 14:05:52

Operations

Yagmur Torun

Entry Subject:

Form:	Controlled Access
Controlled Access Leader:	Mike Wallbank
Others on Controlled Access:	Dave Northacker Yagmur
Run Coordinator Approval:	True
Shifter Informed:	False
Controlled Access Purpose:	replace the N2 cylinder install the spare beamline DAQ machine reset the upstream WC controller check the DAQ front-end electronics cabling for future timing calibration work Mike and I went in a 2nd time to check the heaters in preparation for the week of severe cold ahead
Comments:	Had to shovel our way through on the way in. Dave moved N2 source to a new cylinder. Mike and I installed beamlinedaq01 in the beamline rack. We briefly turned off the SiPM preamp power supplies and moved their power cords to the yellow power strip at the bottom of the rack in order to free up a couple of outlets for the server. We also pulled a pair of network cables from the switch in the network rack to the beamline rack: 1 spare and 1 for the public interface (for which there was already a cable but it was used for the remote power strip) Mike power cycled the controller for the upstream beam WC Turned on the west heater and set it to 65F during the 2nd access

Comments:

Yagmur Torun (torun) 02/06/2021 12:34:36: Forgot to mention Mike also changed the trigger hold-off for the upstream 2ndary beam WC from 10 to 100us.

01/19/2021

DAQ [amhall] Expert

Computing Checklist

8997 01/19/2021 10:22:40

DAQ Expert

Anna Hall

Entry Subject:

Form:	Computing Checklist
:
Is disk space on master okay?:	Yes
Is disk space on the datadisk okay?:	Yes
Are FTS transfers normal?:	Yes
:
Is the page up-to-date?:	Yes
Are there any missing raw files?:	No
Are there any missing database entries?:	No
Are there any missing root files?:	No
Comments::	Actually run 100996 looks like it is missing a single raw file (or two) but it was also a crashed run so I'm not concerned

01/15/2021

Operations [torun]

Controlled Access 2ndary beam WC HV adjustments

8996 01/15/2021 15:02:42

Operations

Yagmur Torun

Entry Subject: 2ndary beam WC HV adjustments

Form:	Controlled Access
Controlled Access Leader:	Yagmur
Others on Controlled Access:	Mike Wallbank
Run Coordinator Approval:	True
Shifter Informed:	False
Controlled Access Purpose:	Lowered target WC HV from 2.4 to 2.35kV Moved the HV PS (Droege) for the upstream chamber to the MCenter Control Rm and connected it to #23 on the HV patch panel (which was unused) Claimed the HV cable from the LArIAT Muon Range Stack that was at #63 on the patch panel at MC7a, moved it to #23 (which was unused) and connected it to the chamber at the other end Set the HV for the upstream chamber to 2.0kV
Comments:	We can now adjust the upstream chamber HV and reset the PS when it trips from the Control Room.

Comments:

Yagmur Torun (torun) 01/15/2021 16:24:44: Made another access and moved to #61 on the patch panels at both ends.

01/14/2021

Operations [torun]

Controlled Access Magnet sweep

8995 01/14/2021 16:56:59

Operations

Yagmur Torun

Entry Subject: Magnet sweep

Form:	Controlled Access
Controlled Access Leader:	Yagmur
Others on Controlled Access:	Dung
Run Coordinator Approval:	True
Shifter Informed:	False
Controlled Access Purpose:	Enclosure sweep before turning on the magnet.
Comments:	Forgot to turn the lights off on my way out, Mike is coming in for another access.

Comments:

Yagmur Torun (torun) 01/14/2021 16:58:18: This access ended around 16:20, by the way.
Yagmur Torun (torun) 01/14/2021 16:58:54: And the magnet has been on since about 16:35.
Yagmur Torun (torun) 01/14/2021 17:21:52: Made another access with Mike, lights are now off, asked for beam.

01/12/2021

Operations [torun] Beam Approved for beam Beam sign-offs in place, see "MCR-Elog 184908":https://www-bd.fnal.gov/Elog/?orEntryId=184908

8994 01/12/2021 13:13:50

Operations Beam

Yagmur Torun

Entry Subject: Approved for beam

Beam sign-offs in place, see

MCR-Elog 184908

01/07/2021

Detector DAQ [lackey32] Expert

Computing Checklist

8993 01/07/2021 10:31:50

Detector DAQ Expert

Teresa Lackey

Entry Subject:

Form:	Computing Checklist
:
Is disk space on master okay?:	Yes
Is disk space on the datadisk okay?:	Yes
Are FTS transfers normal?:	Yes
:
Is the page up-to-date?:	Yes
Are there any missing raw files?:	No
Are there any missing database entries?:	Missing DDActivity1 files Missing Beamline files Missing TBSpill files
Are there any missing root files?:	No
Comments::	I figure we should start doing this checklist again now that FTS is working. DB issues at all detectors, so the db is a few days behind.

12/22/2020

Building [torun] Beam Shield wall complete Good progress today, * The last 2 blocks were installed in the morning * The temporary platform used for bringing in concrete blocks was removed and the rollup door was returned to its previous state * An MCR Ops team came by to inspect the layout and evaluate any changes needed for search & secure procedures and documentation * While they were there, I adjusted the overhead mirror near the rollup door so it shows the entire top of the wall (no need to get up on the ROSIE ladder)

8992 12/22/2020 15:47:56

Building Beam

Yagmur Torun

Entry Subject: Shield wall complete

Good progress today,

The last 2 blocks were installed in the morning
The temporary platform used for bringing in concrete blocks was removed and the rollup door was returned to its previous state
An MCR Ops team came by to inspect the layout and evaluate any changes needed for search & secure procedures and documentation
While they were there, I adjusted the overhead mirror near the rollup door so it shows the entire top of the wall (no need to get up on the ROSIE ladder)

Northwest corner of the wall filled

IMG_6461.JPG

Platform under/outside west rollup door removed

IMG_6466.JPG

12/22/2020

Operations [wallbank] Beam MWPC Powered on the upstream wire chamber I have just installed a 'new' MWPC controller (taken, with permission, from Ewa's area, where she was using it with the test stand) in the upstream location in MC7a. I also turned on the HV and set up the trigger signal to go to the controller. All appears to be functioning correctly; I can communicate with the controller and I see counts on a scalar from cosmics. The controller is ftbfwc04: 1) LOCAL IP : 131.225.176.162 2) G/W IP Addr : 131.225.176.200 3) SUBNET MASK : 255.255.255.0 4) MAC Addr(hex): 00.80.55.00.00.65

8991 12/22/2020 13:10:51

Operations Beam MWPC

Mike Wallbank

Entry Subject: Powered on the upstream wire chamber

I have just installed a 'new' MWPC controller (taken, with permission, from Ewa's area, where she was using it with the test stand) in the upstream location in MC7a. I also turned on the HV and set up the trigger signal to go to the controller.

All appears to be functioning correctly; I can communicate with the controller and I see counts on a scalar from cosmics.

The controller is ftbfwc04:
1) LOCAL IP : 131.225.176.162
2) G/W IP Addr : 131.225.176.200
3) SUBNET MASK : 255.255.255.0
4) MAC Addr(hex): 00.80.55.00.00.65

12/21/2020

Planning [torun] APD Replacement fill cap We have a missing cap on one of the bottom horizontal modules with Al foil covering the fill tube. I brought in a couple of PVC caps and left them on the unused chiller on the west side (in a Home Depot plastic shopping bag). We need the HV off on that module in order to install the cap (or just do it next time the detector is powered off).

8990 12/21/2020 18:55:16

Planning APD

Yagmur Torun

Entry Subject: Replacement fill cap

We have a missing cap on one of the bottom horizontal modules with Al foil covering the fill tube. I brought in a couple of PVC caps and left them on the unused chiller on the west side (in a Home Depot plastic shopping bag). We need the HV off on that module in order to install the cap (or just do it next time the detector is powered off).

12/21/2020

Building [torun] Beam Shield wall installation update * Todd installed the beam pipe in the morning. * The crew wanted feedback on the configuration after they started stacking the steel bricks around the beam pipe. * I noticed the He tube had been pinched between the beam pipe and the most upstream steel brick underneath; talked to Todd who checked the tube (no damage) and pulled it to the side. * The crew finished installing the bricks and put up 3 more blocks afterward. * The wall should be finished tomorrow morning: one block needs to be moved a couple of inches to its final position and there are 2 more blocks to go into the northwest corner. * The top of the wall is below the top of ROSIE and clearly visible from the ladder on the downstream west side of ROSIE; the view looking up at the ceiling mirror on the west side from the floor spans most of the top of the wall and probably could be adjusted to see the rest.

8989 12/21/2020 18:43:25

Building Beam

Yagmur Torun

Entry Subject: Shield wall installation update

Todd installed the beam pipe in the morning.
The crew wanted feedback on the configuration after they started stacking the steel bricks around the beam pipe.
I noticed the He tube had been pinched between the beam pipe and the most upstream steel brick underneath; talked to Todd who checked the tube (no damage) and pulled it to the side.
The crew finished installing the bricks and put up 3 more blocks afterward.
The wall should be finished tomorrow morning: one block needs to be moved a couple of inches to its final position and there are 2 more blocks to go into the northwest corner.
The top of the wall is below the top of ROSIE and clearly visible from the ladder on the downstream west side of ROSIE; the view looking up at the ceiling mirror on the west side from the floor spans most of the top of the wall and probably could be adjusted to see the rest.

East

IMG_6457.JPG

Center

IMG_6458.JPG

West

IMG_6455.JPG

Missing northwest corner

IMG_6454.JPG

View from ROSIE ladder

IMG_6460.JPG

View through west mirror

IMG_6459.JPG

12/18/2020

DCS [asutton] DCS Expert Trying another thing for the DCS R17 issues Changing the setup scripts on the novadcs-test-01 to for R16 rather than use the daq-config setting novadaq@novadcs-test-01:/home/novadaq/bin/start-opicssdcs.sh novadcs@novadcs-test-01:~/bash_profiles/bashrc-novadcs These have a few places where I've forced R16

8988 12/18/2020 09:58:04

DCS DCS Expert

Andrew Sutton

Entry Subject: Trying another thing for the DCS R17 issues

Changing the setup scripts on the novadcs-test-01 to for R16 rather than use the daq-config setting

novadaq@novadcs-test-01:/home/novadaq/bin/start-opicssdcs.sh

novadcs@novadcs-test-01:~/bash_profiles/bashrc-novadcs

These have a few places where I've forced R16

12/16/2020

Detector DAQ [asutton] DAQ Software Expert Another stab at R17 DCM weirdness Poking a bit more at the NovaDaqDcs package in R17 I found a few more interesting files with differences: NovaDaqDcs/epics/dbd/regIOC.dbd R16: < choice(menuConverttypeSdegF,"FEBTempADC2degreeC") < choice(menuConverttypeSdegC,"TECCadc2degreeC") --- R17: > choice(menuConverttypeSdegF,"typeSdegF") > choice(menuConverttypeSdegC,"typeSdegC") NovaDaqDcs/epics/regIOCApp/src/O.Common/regIOC.dbd R16: < choice(menuConverttypeSdegF,"FEBTempADC2degreeC") < choice(menuConverttypeSdegC,"TECCadc2degreeC") --- R17: > choice(menuConverttypeSdegF,"typeSdegF") > choice(menuConverttypeSdegC,"typeSdegC") NovaDaqDcs/epics/regIOCApp/src/O.linux-x86_64/regIOC.dbd.d R16: < ../O.Common/regIOC.dbd : ../../../dbd/menuConvert.dbd --- R17: > ../O.Common/regIOC.dbd : /nova/ups/epics/v3_14_12/dbd/menuConvert.dbd I'm pretty sure that these are auto generated upon building and I'm not sure what changed to cause the difference. I think the path difference in NovaDaqDcs/epics/regIOCApp/src/O.linux-x86_64/regIOC.dbd.d caused the knock on changes to the other two .dbd files but I can't find any reason for the path change there (might just need to look harder).

8987 12/16/2020 10:44:02

Detector DAQ DAQ Software Expert

Andrew Sutton

Entry Subject: Another stab at R17 DCM weirdness

Poking a bit more at the NovaDaqDcs package in R17 I found a few more interesting files with differences:

NovaDaqDcs/epics/dbd/regIOC.dbd
R16:
< choice(menuConverttypeSdegF,"FEBTempADC2degreeC")
< choice(menuConverttypeSdegC,"TECCadc2degreeC")
---
R17:
> choice(menuConverttypeSdegF,"typeSdegF")
> choice(menuConverttypeSdegC,"typeSdegC")

NovaDaqDcs/epics/regIOCApp/src/O.Common/regIOC.dbd
R16:
< choice(menuConverttypeSdegF,"FEBTempADC2degreeC")
< choice(menuConverttypeSdegC,"TECCadc2degreeC")
---
R17:
> choice(menuConverttypeSdegF,"typeSdegF")
> choice(menuConverttypeSdegC,"typeSdegC")

NovaDaqDcs/epics/regIOCApp/src/O.linux-x86_64/regIOC.dbd.d
R16:
< ../O.Common/regIOC.dbd : ../../../dbd/menuConvert.dbd
---
R17:
> ../O.Common/regIOC.dbd : /nova/ups/epics/v3_14_12/dbd/menuConvert.dbd

I'm pretty sure that these are auto generated upon building and I'm not sure what changed to cause the difference. I think the path difference in NovaDaqDcs/epics/regIOCApp/src/O.linux-x86_64/regIOC.dbd.d caused the knock on changes to the other two .dbd files but I can't find any reason for the path change there (might just need to look harder).

Comments:

Andrew Sutton (asutton) 12/16/2020 10:57:02

I’ve altered the R17 versions of these files which are used when running the regIOC. This actually runs on the dcs machine and the dcmIOC stuff doesn’t look changed at all.

Andrew Sutton (asutton) 12/16/2020 11:00:20

Going to stop the run, roll forward to R17, reboot dcm2, try to cool.

I don’t think this will be enough so once that doesn’t work then I’ll restart regIOC on novadcs-test-01, restart the CSS gui (just for good measure), then reboot dcm3, and try to cool.

Andrew Sutton (asutton) 12/16/2020 11:26:19

Still no good. The DCM is staying in alarm after cooling claiming APD LOLO

Andrew Sutton (asutton) 12/16/2020 11:27:33

Undid all file changes, rolled back to R16, restarted regIOC, restarted CSS, restarted dcm2, cooled perfectly, bringing back run

12/15/2020

Detector DAQ [lackey32] DAQ Software Expert Yet another attempt at building R17. Andrew S. noticed yesterday that there is some weirdness in some of the files in the build (particularly in NovaDaqDcs) where they have the full path /data/nova/novadaq... rather than following the symlink to just /nova/novadaq. This could be causing problems for any machine that isn't test-mgr-01, which only has the mount for /nova. I'm going to attempt another full clean build of R17, but do it on test-db this time instead of test-mgr-01.

8986 12/15/2020 11:14:03

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Yet another attempt at building R17.

Andrew S. noticed yesterday that there is some weirdness in some of the files in the build (particularly in NovaDaqDcs) where they have the full path /data/nova/novadaq... rather than following the symlink to just /nova/novadaq. This could be causing problems for any machine that isn't test-mgr-01, which only has the mount for /nova. I'm going to attempt another full clean build of R17, but do it on test-db this time instead of test-mgr-01.

Comments:

Teresa Lackey (lackey32) 12/15/2020 12:31:33: Seeing some build errors go past, missing packages on test-db. Did a yum install libconfuse. Not sure if I’m missing any others, but once this build finishes I’ll try running it again (without a clean in the middle) to see if it fixes the errors.
Teresa Lackey (lackey32) 12/15/2020 12:32:32: Also yum install gcc-c++ for the daqdcs build.
Teresa Lackey (lackey32) 12/15/2020 12:43:15: Second rebuild had no obvious errors. Just to be safe I’m going to do a full clean rebuild again now that the correct packages are installed.
Teresa Lackey (lackey32) 12/15/2020 14:09:19: Build complete. Logs in .cron/log/novadaq-test-db/*Tue1245.log.
Teresa Lackey (lackey32) 12/15/2020 14:16:04: Errors in building NovaMessageLogger.lib (package rmscore does not exist). Additional errors for ShmMilliBlock.lib (cannot find -lShmRdWr) and TimeChecker.cc (no rule to make target -lShmMilliBlock) in default and opt builds.
Teresa Lackey (lackey32) 12/15/2020 14:23:36: A second build (without clean in the middle) seems to have resolved the errors in the previous comment. I think this is at a point where we can test again.

12/14/2020

Operations [torun] TOF TOF ORC heads-up While Raymond Lewis was at MC7, I showed him the changes in the hall corresponding to the updates we intend to make to the ORCs * locations of TOF detectors * rerouted cables * additional HV PS intended for one of the upstream muon halo paddles

8985 12/14/2020 21:55:16

Operations TOF

Yagmur Torun

Entry Subject: TOF ORC heads-up

While Raymond Lewis was at MC7, I showed him the changes in the hall corresponding to the updates we intend to make to the ORCs

locations of TOF detectors
rerouted cables
additional HV PS intended for one of the upstream muon halo paddles

12/14/2020

Operations [torun] Expert MWPC WC readout hardware update After Mike got the OK from FTBF to swipe TDC cards from upstream (LArIAT) chambers, we started moving on this today * I removed the bottom east card from the 2nd most upstream chamber in MC7a (labeled M) and connected it to the ethernet cable for the bottom west one on our WC3 (the one Andrew has identified as suspect) * Andrew ID'ed the card through the controller, it reports the same firmware version but a different checksum compared to most of the cards we're using; not clear it needs a new firmware flash; Mike has contacted Terry K for guidance

8984 12/14/2020 21:46:43

Operations Expert MWPC

Yagmur Torun

Entry Subject: WC readout hardware update

After Mike got the OK from FTBF to swipe TDC cards from upstream (LArIAT) chambers, we started moving on this today

I removed the bottom east card from the 2nd most upstream chamber in MC7a (labeled M) and connected it to the ethernet cable for the bottom west one on our WC3 (the one Andrew has identified as suspect)
Andrew ID’ed the card through the controller, it reports the same firmware version but a different checksum compared to most of the cards we’re using; not clear it needs a new firmware flash; Mike has contacted Terry K for guidance

Chamber M

IMG_6400.JPG

TDC card removed

IMG_6405.JPG

and connected to WC3

IMG_6416.JPG

12/14/2020

Planning [torun] Preparing for detector tilt - part 2 Did a walk-through with Jim Kilmer in the morning * in addition to the sides which we have already cleared, they need access to the bottom of the detector in order to place jacks and shims underneath the center I-beam * I moved miscellaneous components next to or stuffed under the detector blocks (oil barrels and carboys, fill cart and associated plumbing, tall ladder, etc.) to the edges of the containment mat; do not put anything inside the mat * we still need to worry about not puncturing the mat, Jim is aware * I wanted more through bolts installed connecting the I-beams of the 2 blocks for additional safety margin during lifts; closer inspection shows this is only possible for 2 holes on the center beams

8983 12/14/2020 21:23:29

Planning

Yagmur Torun

Entry Subject: Preparing for detector tilt - part 2

Did a walk-through with Jim Kilmer in the morning

in addition to the sides which we have already cleared, they need access to the bottom of the detector in order to place jacks and shims underneath the center I-beam
I moved miscellaneous components next to or stuffed under the detector blocks (oil barrels and carboys, fill cart and associated plumbing, tall ladder, etc.) to the edges of the containment mat; do not put anything inside the mat
we still need to worry about not puncturing the mat, Jim is aware
I wanted more through bolts installed connecting the I-beams of the 2 blocks for additional safety margin during lifts; closer inspection shows this is only possible for 2 holes on the center beams

Downstream end

ds block bottom east

ds block bottom west

us block bottom west

us block bottom east

us end west

us end east

east aisle

12/14/2020

Building [torun] Beam MC7a muon shielding status - Wall configuration := talked to mechanical crew, Jim Kilmer and Raymond Lewis in the morning; Jim apparently had to make a change to the beam pipe hole due to limited supply of L-blocks; possible to move wall east by about 6" in that configuration (with respect to the old design drawing) by trimming one of the unistrut shelves recently installed to support JGG power cables but not without violating the minimum east aisle width - East aisle := the racks on the east side taking up about a foot of aisle width were supposed to be removed weeks ago together with the rest; talked to Todd who (with help from Jim W) disconnected fans and smoke detectors on those racks and pulled them out; I moved the remaining loose cables and whatnot out of the way, problem solved; met again with Jim, Raymond and the mechanical crew and agreed to place the east edge of the shield wall 99" east of the beam centerline, (6" past ROSIE, roughly lined up with the edge of the concrete pad underneath) - Installation := the 3 blocks installed Thursday were pulled today to cut and install the missing plywood base panels; 1 block was reinstalled and aligned by the end of the day, the rest should go smoothly

8982 12/14/2020 20:46:41

Building Beam

Yagmur Torun

Entry Subject: MC7a muon shielding status

Wall configuration: talked to mechanical crew, Jim Kilmer and Raymond Lewis in the morning; Jim apparently had to make a change to the beam pipe hole due to limited supply of L-blocks; possible to move wall east by about 6” in that configuration (with respect to the old design drawing) by trimming one of the unistrut shelves recently installed to support JGG power cables but not without violating the minimum east aisle width
East aisle: the racks on the east side taking up about a foot of aisle width were supposed to be removed weeks ago together with the rest; talked to Todd who (with help from Jim W) disconnected fans and smoke detectors on those racks and pulled them out; I moved the remaining loose cables and whatnot out of the way, problem solved; met again with Jim, Raymond and the mechanical crew and agreed to place the east edge of the shield wall 99” east of the beam centerline, (6” past ROSIE, roughly lined up with the edge of the concrete pad underneath)
Installation: the 3 blocks installed Thursday were pulled today to cut and install the missing plywood base panels; 1 block was reinstalled and aligned by the end of the day, the rest should go smoothly

East aisle with the offending racks

IMG_6304.JPG

Racks removed

IMG_6397.JPG

Aisle clear

IMG_6398.JPG

Wall at the end of last week

IMG_6350(1).JPG

Wall at the end of today

IMG_6401.JPG

12/11/2020

Operations [torun] Today's progress * Dave hooked up a CO2 cylinder and started gas flow into the Ckov line * Dave also unrolled the containment mat outside * I put up the orange fence between WC4 and Ckov

8981 12/11/2020 18:27:53

Operations

Yagmur Torun

Entry Subject: Today's progress

Dave hooked up a CO2 cylinder and started gas flow into the Ckov line
Dave also unrolled the containment mat outside
I put up the orange fence between WC4 and Ckov

CO2 cylinder

IMG_6369.JPG

Gas panel

IMG_6375.JPG

Containment mat outside

IMG_6367.JPG

Fence around TOF

IMG_6374.JPG

12/11/2020

Operations [wallbank]

Beamline Start Run Post-ToF-move cosmic run (US)

8980 12/11/2020 13:54:54

Operations

Mike Wallbank

Entry Subject: Post-ToF-move cosmic run (US)

Form:	Beamline Start Run
Run number:	3596
Beamline detectors:	Trigger board Digitizer TDU
Trigger conditions:	ToF-US 3-of-4
Magnet Current:	Off
Magnet Current Polarity:	Off
MCenter Beam On?:	False
Digitizer frequency:	2.5 GHz
Comments:	The arm in the US position is now the PMT arm which was moved from the DS location yesterday.

Comments:

Mike Wallbank (wallbank) 12/11/2020 16:29:10: Run ended

12/11/2020

Operations [wallbank]

Beamline Start Run Post-ToF-move cosmic run

8979 12/11/2020 11:35:44

Operations

Mike Wallbank

Entry Subject: Post-ToF-move cosmic run

Form:	Beamline Start Run
Run number:	3595
Beamline detectors:	Trigger board Digitizer TDU
Trigger conditions:	ToF-DS-1 (PMT) 3-of-4
Magnet Current:	Off
Magnet Current Polarity:	Off
MCenter Beam On?:	False
Digitizer frequency:	2.5 GHz
Comments:	Arm is in its new location, between MWPC4 and Ckov.

Comments:

Mike Wallbank (wallbank) 12/11/2020 13:51:22: ending run to take some data with the ToF-US module

12/10/2020

Building [torun] MC7 rad fence and door fixed The east side fence for the batting cage is back up with stainless clamps. Also a photo of the new MC7 upstream west door with the lock and signage installed.

8978 12/10/2020 22:10:50

Building

Yagmur Torun

Entry Subject: MC7 rad fence and door fixed

The east side fence for the batting cage is back up with stainless clamps. Also a photo of the new MC7 upstream west door with the lock and signage installed.

Batting cage east fence

IMG_6347.JPG

Batting cage west fence

IMG_6353.JPG

MC7 upstream west door

IMG_6339.JPG

12/10/2020

Operations [torun] Cherenkov MWPC TOF Beamline detectors back on Todd carried out the "Big ToF Rearrangement" in the morning * removed beam pipe section between WC4 and Ckov * moved the thick-slab PMT-TOF from the upstream position to just upstream of the Ckov; it's now about 34.5" downstream of WC4 * moved the thin-slab PMT-TOF from next to the NOvA detector to the upstream position Mike and I reconnected the TOFs at their new locations in the afternoon * I pulled the DS-TOF signal and HV cables back along the cable trays to just above the rack and routed them overhead to the Ckov support block; this wasn't easy as the HV cables were being crushed by the heavy gauge detector power cables at the downstream end of the cable tray * we left the HV cables connected to the same cow channels and changed the output voltages; this requires an update to the synoptic monitor settings * we also checked the length of the PMT signal pigtails for the two TOFs and they seem to be the same (about 5') All 3 TOFs, WCs and Ckov are now back on. I noticed scratches on the downstream window of WC4, we should cover the windows whenever working near the chambers.

8977 12/10/2020 21:55:05

Operations Cherenkov MWPC TOF

Yagmur Torun

Entry Subject: Beamline detectors back on

Todd carried out the “Big ToF Rearrangement” in the morning

removed beam pipe section between WC4 and Ckov
moved the thick-slab PMT-TOF from the upstream position to just upstream of the Ckov; it’s now about 34.5” downstream of WC4
moved the thin-slab PMT-TOF from next to the NOvA detector to the upstream position
Mike and I reconnected the TOFs at their new locations in the afternoon
I pulled the DS-TOF signal and HV cables back along the cable trays to just above the rack and routed them overhead to the Ckov support block; this wasn’t easy as the HV cables were being crushed by the heavy gauge detector power cables at the downstream end of the cable tray
we left the HV cables connected to the same cow channels and changed the output voltages; this requires an update to the synoptic monitor settings
we also checked the length of the PMT signal pigtails for the two TOFs and they seem to be the same (about 5’)
All 3 TOFs, WCs and Ckov are now back on.
I noticed scratches on the downstream window of WC4, we should cover the windows whenever working near the chambers.

WC4 ds window

IMG_6344.JPG

WC4 ds window (less reflection)

TOF cow b4

TOF cow after

DS-TOF one

DS-TOF two

DS-TOF three

12/10/2020

8976 12/10/2020 21:54:19

Operations Cherenkov MWPC TOF

Yagmur Torun

Entry Subject: Beamline detectors back on

Todd carried out the “Big ToF Rearrangement” in the morning

removed beam pipe section between WC4 and Ckov
moved the thick-slab PMT-TOF from the upstream position to just upstream of the Ckov; it’s now about 34.5” downstream of WC4
moved the thin-slab PMT-TOF from next to the NOvA detector to the upstream position
Mike and I reconnected the TOFs at their new locations in the afternoon
I pulled the DS-TOF signal and HV cables back along the cable trays to just above the rack and routed them overhead to the Ckov support block; this wasn’t easy as the HV cables were being crushed by the heavy gauge detector power cables at the downstream end of the cable tray
we left the HV cables connected to the same cow channels and changed the output voltages; this requires an update to the synoptic monitor settings
we also checked the length of the PMT signal pigtails for the two TOFs and they seem to be the same (about 5’)
All 3 TOFs, WCs and Ckov are now back on.
I noticed scratches on the downstream window of WC4, we should cover the windows whenever working near the chambers.

WC4 ds window

IMG_6344.JPG

WC4 ds window (less reflection)

TOF cow b4

TOF cow after

DS-TOF one

DS-TOF two

DS-TOF three

12/09/2020

Operations [wallbank] Cherenkov MWPC TOF Powering down beamline detectors, disconnecting PMTs In preparation for the Big ToF Rearrangement tomorrow morning, I am removing power from all beamline detectors and disconnecting the PMTs (since these arms will be moved). We will hopefully be able to bring things back tomorrow.

8975 12/09/2020 17:26:56

Operations Cherenkov MWPC TOF

Mike Wallbank

Entry Subject: Powering down beamline detectors, disconnecting PMTs

In preparation for the Big ToF Rearrangement tomorrow morning, I am removing power from all beamline detectors and disconnecting the PMTs (since these arms will be moved).

We will hopefully be able to bring things back tomorrow.

12/09/2020

Detector DAQ [lackey32] DAQ Software Expert Clean build of R17. We still had issues with switching to R17. This time the DCMs started their IOC just fine, but after cooling everything stayed in alarm on the CSS GUI despite being cold. I'm going to try a completely clean build of R17 using /home/novadaq/.cron/clean_base_release_all, and then another update_base_release_all.

8974 12/09/2020 12:40:49

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Clean build of R17.

We still had issues with switching to R17. This time the DCMs started their IOC just fine, but after cooling everything stayed in alarm on the CSS GUI despite being cold. I'm going to try a completely clean build of R17 using /home/novadaq/.cron/clean_base_release_all, and then another update_base_release_all.

Comments:

Teresa Lackey (lackey32) 12/09/2020 13:03:24

./cron/clean_base_release_all R17_00_00 done.
From a fresh login, did

envresetarch
export CXXFLAGS=”-include$HOME/nova_sl6_to_sl7_fix.hpp -pthread” LC_ALL=C LANG=en_US
.cron/update_base_release_all —no-update R17_00_00

Teresa Lackey (lackey32) 12/09/2020 13:19:19

The above should read

envresetarch —uname-2.6

Teresa Lackey (lackey32) 12/09/2020 14:01:46

Build is complete. I don’t want to take things down again to test today, but R17 is ready to test again when we have the chance. I’m felling a little better about this build. We don’t have a mix of 2.6 and 3.1 binary directories, and the daqdcs build worked the first time through.

12/09/2020

Detector DAQ [lackey32] DAQ Software Expert Stopping detector run and rebooting detector to test dependency on old manager machines. Seth turned off master, trigger, runcontrol, and datadisk at 9:30 this morning. I'm going to test that we don't have any dependencies on these machines and also switch to R17. More details will be on the main NOvA ECL "ECL-181594":https://dbweb8.fnal.gov:8443/ECL/nova/E/show?e=181594

8973 12/09/2020 09:55:44

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Stopping detector run and rebooting detector to test dependency on old manager machines.

Seth turned off master, trigger, runcontrol, and datadisk at 9:30 this morning. I’m going to test that we don’t have any dependencies on these machines and also switch to R17. More details will be on the main NOvA ECL ECL-181594

12/09/2020

Beamline DAQ [asutton] Expert MWPC WC Tests Going to be running WC tests throughout today to address the issues with WC3 having no hits on most of the west half-plane in X. reference in the NOvA ECL: https://dbweb8.fnal.gov:8443/ECL/nova/E/show?e=181592

8972 12/09/2020 09:23:02

Beamline DAQ Expert MWPC

Andrew Sutton

Entry Subject: WC Tests

Going to be running WC tests throughout today to address the issues with WC3 having no hits on most of the west half-plane in X.

reference in the NOvA ECL: https://dbweb8.fnal.gov:8443/ECL/nova/E/show?e=181592

Comments:

Andrew Sutton (asutton) 12/09/2020 09:25:42

Turning the TDC pulser on at 203dec (~5 Hz). I’m pretty sure that this actively pulses the ASDQ card channels. We should not see hits on the “dead” channels if my understanding is correct.

If we do have hits then I’ll have to think a bit harder

Andrew Sutton (asutton) 12/09/2020 13:15:03

Got absolutely no where with this. Couldn’t get a run going with the pulser firing. Have no clue why.

12/07/2020

Detector DAQ [lackey32] Expert OnMon/EVD Running onmon prod with gdb The OnMon producer keeps crashing on subrun rollover when it tries closing the root file. I'm starting up the producer in a terminal using gdb to see if I can get some more information on the next crash. Viewer still running from the icon.

8971 12/07/2020 14:49:36

Detector DAQ Expert OnMon/EVD

Teresa Lackey

Entry Subject: Running onmon prod with gdb

The OnMon producer keeps crashing on subrun rollover when it tries closing the root file. I'm starting up the producer in a terminal using gdb to see if I can get some more information on the next crash. Viewer still running from the icon.

Comments:

Teresa Lackey (lackey32) 12/07/2020 15:22:58: Well that didn’t help at all – there’s no stack.

12/07/2020

Detector DAQ [lackey32] DAQ Software Expert Fixed R17 daqdcs build. Last week we noticed the xcompile-opt-daqdcs build of R17 was messed up, leading to issues with the CSS GUI, since the startIOC scripts were missing from the LinuxPPC-GCC-maxopt bin (so DCMs weren't restarting the IOC on boot). The errors in the logfile from the build were just complaining about 'g++ command not found', so I did yum install gcc-c++ and ran the build again. The scripts are now in the correct bin folder, so I think everything worked. We'll need to do a full test later this week once we've had enough continuous running to satisfy MCR.

8970 12/07/2020 14:24:33

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Fixed R17 daqdcs build.

Last week we noticed the xcompile-opt-daqdcs build of R17 was messed up, leading to issues with the CSS GUI, since the startIOC scripts were missing from the LinuxPPC-GCC-maxopt bin (so DCMs weren't restarting the IOC on boot). The errors in the logfile from the build were just complaining about 'g++ command not found', so I did

yum install gcc-c++

and ran the build again. The scripts are now in the correct bin folder, so I think everything worked. We'll need to do a full test later this week once we've had enough continuous running to satisfy MCR.

12/05/2020

Detector DAQ [lackey32] DAQ Software Expert Updated msglogger node in kill partition. Realized that the msglogger node in kill partition was still mgr-01, so kill partition wasn't killing any of the msglogger managers. It's now pointing to mgr-02.

8969 12/05/2020 03:30:08

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Updated msglogger node in kill partition.

Realized that the msglogger node in kill partition was still mgr-01, so kill partition wasn't killing any of the msglogger managers. It's now pointing to mgr-02.

Comments:

Teresa Lackey (lackey32) 12/15/2020 11:58:45: Just went through the Kill Partition script and removed all references to our old manager nodes so the script doesn’t have to hang on trying to access each of those.

12/04/2020

Operations [wallbank] Powering up all systems and starting run to hand over to the NOvA shifter We are turning on HV on the Cherenkov and the NOvA detector and starting a run. Aim to hand over to the NOvA shifter today.

8968 12/04/2020 10:45:34

Operations

Mike Wallbank

Entry Subject: Powering up all systems and starting run to hand over to the NOvA shifter

We are turning on HV on the Cherenkov and the NOvA detector and starting a run. Aim to hand over to the NOvA shifter today.

Comments:

Yagmur Torun (torun) 12/04/2020 11:08:25: Detector Run 100908 in progress

12/03/2020

Detector DAQ [wallbank] APD DCM Expert Powering down detector and testing power supply alarms We're finishing our running for the day so I will end the run and power down the detector. Tom and I are also planning to test the power supply alarms which he has set up as we power things down.

8967 12/03/2020 16:27:00

Detector DAQ APD DCM Expert

Mike Wallbank

Entry Subject: Powering down detector and testing power supply alarms

We're finishing our running for the day so I will end the run and power down the detector.

Tom and I are also planning to test the power supply alarms which he has set up as we power things down.

Comments:

Mike Wallbank (wallbank) 12/03/2020 16:31:29: Turned off HV. Tom got a text! It works :)
Mike Wallbank (wallbank) 12/03/2020 16:34:04: Turned off LV, same thing. Sounds like there’ll be a lot of emails if something ever goes wrong with the TB power supplies…

12/03/2020

Detector DAQ [lackey32] DSO/Pedestal Scan

DSO Scan Report Scan for DCM2 after FEB 60 was replaced yesterday.

8966 12/03/2020 13:30:08

Detector DAQ DSO/Pedestal Scan

Teresa Lackey

Entry Subject: Scan for DCM2 after FEB 60 was replaced yesterday.

Form:	DSO Scan Report
Detector is cool:	True
Directory name (dcm 1+2):	2020-12-03_13_21_50
Directory name (dcm 3):	NA
Configuration Name:	TestBeam_AllTrigs_FullDetector
Number of channels for dcms 1 + 2:	1/58/1856
Number of channels for dcm 3:	NA
:	True
Comments?:	FEB 50 had the most obvious differences, this is the one that was swapped. Since 59 and 60 had swapped cables, there are also some noticeable differences on 59.
Detector is still cool:	True

thresh_dcm2.png

threshDif..2.png

Comments:

Teresa Lackey (lackey32) 12/03/2020 14:01:51: 60, not 50.

12/03/2020

Detector DAQ [lackey32] DAQ Software Expert Rolling back to R16, DSO scan for dcm 2, starting a run. There were issues with the xcompile-opt-daqdcs build in R17, so I'm rolling back to R16 while we investigate it. I'll do a DSO scan on dcm2 since it has a new FEB, and then I'll get a run going. Hopefully EVD/OnMon/DSOResultsViewer are all setup correctly now, but this is the first test of them now that test-db is SL7.

8965 12/03/2020 13:05:11

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Rolling back to R16, DSO scan for dcm 2, starting a run.

There were issues with the xcompile-opt-daqdcs build in R17, so I'm rolling back to R16 while we investigate it. I'll do a DSO scan on dcm2 since it has a new FEB, and then I'll get a run going. Hopefully EVD/OnMon/DSOResultsViewer are all setup correctly now, but this is the first test of them now that test-db is SL7.

Comments:

Teresa Lackey (lackey32) 12/03/2020 13:07:04

Looks like /nova/config/TestBeam/daq-operations.cfg was already rolled back to point to the TB16_00_00 version. Turning on LV, then HV.

Teresa Lackey (lackey32) 12/03/2020 13:12:43

dcm1 3.5 didn’t come on. Trying again with its individual power-on script worked, just waiting to see if it’ll decide soon that it has a TEC. If not, I’ll power cycle 1.

Teresa Lackey (lackey32) 12/03/2020 13:17:46

LV and HV on. Cooling detector.

Teresa Lackey (lackey32) 12/03/2020 13:22:28

Detector cold. Doing DSO scan for dcm2.

Teresa Lackey (lackey32) 12/03/2020 13:38:19

Run going, need to make a testrel for EVD/OnMon to put in a quick bug fix.

Teresa Lackey (lackey32) 12/03/2020 13:52:02

Nevermind, didn’t need a testrel. I looked at the error again and realized we just needed

yum install libglvnd-opengl openssl-devel

Teresa Lackey (lackey32) 12/03/2020 14:01:15

EVD/OnMon/DSOResultsViewer icons all work on SL7.

Teresa Lackey (lackey32) 12/03/2020 14:13:54

OnMon producer crashed. This was happening previously at each subrun rollover, but fixed itself on the next run. If it crashes again, I’ll probably just restart the run since it’s pretty quick to do.

Teresa Lackey (lackey32) 12/03/2020 14:23:48

Restarting the run with the hopes OnMon will cooperate.

Teresa Lackey (lackey32) 12/03/2020 14:46:10

OnMon crashed again, but then we got a datalogger unresponsive error anyways. Restarting the run again.

Teresa Lackey (lackey32) 12/03/2020 15:19:20

OnMon producer is still crashing each time it closes a file (so after subrun rollover). I’m not really sure why, or how to solve it.

Teresa Lackey (lackey32) 12/03/2020 15:30:24

Ah. Just remembered high beamline trigger rates have caused issues for OnMon in the past. Looks like EVD is also crashing. I checked the trigger rates, and Beamline doesn’t look too bad, but DDActivity was high. Looks like Kill Parition didn’t properly kill Global Trigger. I killed the older one and hopefully things will keep running.

12/02/2020

Operations [torun] Preparing for detector tilt Tom and I did some additional cleanup around the detector to make access easier for the mechanical crew for raising the east side * routed the loose RTD cables hanging below the frame to the upstream face and parked them on the east side * moved the mini-rack (muon paddle front end) at the southeast corner upstream and parked it under the gas panel; disconnected the signal cables from the back of the lemo-to-BNC patch panel (channels 1 through 4 top to bottom)

8964 12/02/2020 17:54:03

Operations

Yagmur Torun

Entry Subject: Preparing for detector tilt

Tom and I did some additional cleanup around the detector to make access easier for the mechanical crew for raising the east side

routed the loose RTD cables hanging below the frame to the upstream face and parked them on the east side
moved the mini-rack (muon paddle front end) at the southeast corner upstream and parked it under the gas panel; disconnected the signal cables from the back of the lemo-to-BNC patch panel (channels 1 through 4 top to bottom)

12/02/2020

Detector DAQ [wallbank] APD Expert FEB APD/FEB swaps On Monday, we did the first lot of APD/FEB swaps. Tom's report: https://docs.google.com/spreadsheets/d/1InBr0cJbPE2CITzb6eqFIH9XTBeeAHYPa4z698Vb-QI/edit#gid=0 I changed 3 APDs and 3 FEBs today. 2 APDs were due to faulty cooling, 1 is hopefully just a duplicate serial number issue. Today we are doing one final FEB swap. Powering down the detector now. FEB 5-1-2:59

8963 12/02/2020 12:54:51

Detector DAQ APD Expert FEB

Mike Wallbank

Entry Subject: APD/FEB swaps

On Monday, we did the first lot of APD/FEB swaps. Tom's report:

https://docs.google.com/spreadsheets/d/1InBr0cJbPE2CITzb6eqFIH9XTBeeAHYPa4z698Vb-QI/edit#gid=0

I changed 3 APDs and 3 FEBs today. 2 APDs were due to faulty cooling, 1 is hopefully just a duplicate serial number issue.

Today we are doing one final FEB swap. Powering down the detector now.

FEB 5-1-2:59

Comments:

Mike Wallbank (wallbank) 12/02/2020 13:07:32: Detector powered down. Tom is doing the swap.
Mike Wallbank (wallbank) 12/02/2020 13:25:36: Swap done. We noticed cables 59 and 60 were connected the wrong way around, so we swapped them. So the new FEB is now 60, which is correct for its location (though it was previously connected to port 59).

12/02/2020

Detector DAQ [lackey32] DAQ Software Expert Turning detector on and trying to switch to R17_00_00 Turning things on for the day. Going to try switching to R17. I need to setup the testrel first.

8962 12/02/2020 06:51:42

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Turning detector on and trying to switch to R17_00_00

Turning things on for the day. Going to try switching to R17. I need to setup the testrel first.

Comments:

Teresa Lackey (lackey32) 12/02/2020 06:54:32

Created testRel_TB17_00_00. Copied DatabaseUtils, NovaGlobalTrigger, NovaRunControl, PedestalDataRunner, and TDUControl from testRel_TB16_00_00_ELFRCTest. Some of these had uncommitted changes, Run Control was committed, but wouldn’t have made it into R17.

Prior to building, did:
envresetarch —uname-2.6
and
export CXXFLAGS=”-include$HOME/nova_sl6_to_sl7_fix.hpp -pthread” LC_ALL=C LANG=en_US

then build4 allpackages – building now.

Teresa Lackey (lackey32) 12/02/2020 06:58:01

Created /nova/config/TestBeam/daq-operations.cfg_TB17_00_00. Once the test release finishes building I’ll link daq-operations.cfg to the new one and try getting a run going. Turning on LV and HV now.

Teresa Lackey (lackey32) 12/02/2020 07:01:25

Cooling detector.

Teresa Lackey (lackey32) 12/02/2020 07:08:41

Build had some errors for TDU Control in all 4 builds, plus some additional root-config and g++ command not found errors in the opt-xcompile build.

TDU Control error is /home/novadaq/testRel_TB17_00_00/TDUControl/cxx/src/TDUControlClient.cpp:2846:8: error: ‘v102_25’ was not declared in this scope

but I can see it defined in TDUFirmwareConstants.h.

Going to just try starting things up anyways.

Teresa Lackey (lackey32) 12/02/2020 07:22:09

It can’t find the rmServer binary. I’m going to roll back to R16 and debug the build.

Teresa Lackey (lackey32) 12/02/2020 07:27:47

Looks like everything ended up in the Linux3.1-GCC bin rather than the Linux2.6-GCC bin. Trying to figure out how this was done on the other detectors. On ND the 3.1 bin doesn’t have the nova stuff while the 2.6 one does.

Teresa Lackey (lackey32) 12/02/2020 07:53:25

So far it looks like this build is putting binaries in the correct location. I have no idea what I did differently this time compared to last time.

Teresa Lackey (lackey32) 12/02/2020 08:01:37

Resource manager started in R17. Going to try starting a run.

Teresa Lackey (lackey32) 12/02/2020 08:04:18

dcmapplication not starting. There’s an error in the logfile about not being able to execute the binary. I recall this happening on the other detectors, going to see if I can find the solution in the other ECL.

Teresa Lackey (lackey32) 12/02/2020 08:07:33

Ah, copying the srt_environment file had to be done again after the new build. (Steps towards the bottom of https://cdcvs.fnal.gov/redmine/projects/novadaq-releases/wiki/Example_of_creating_and_testing_new_base_release_)

Teresa Lackey (lackey32) 12/02/2020 08:12:45

Run going with R17. I’ve brought up EVD and OnMon for now, but whenever Seth gets to updating test-db today those will disappear.

Teresa Lackey (lackey32) 12/02/2020 10:19:20

Seth is starting work on test-db. I’m going to stop OnMon and EVD. I’ll leave the run going an keep an eye on trigger scalars and the nearline plots until we bring things down for maintenance.

Teresa Lackey (lackey32) 12/02/2020 10:20:20

I won’t keep an eye on the nearline actually, since that also runs on test-db.

Mike Wallbank (wallbank) 12/02/2020 12:47:56

Stopping run now to swap out an FEB.

Andrew Sutton (asutton) 12/02/2020 15:35:52

The DCS machine doesn’t like that we’ve moved to SL7. When trying to run the DCSConfigEditor (which is routed through novadcs-test-01) it complains with:

DCSConfigEditor: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by DCSConfigEditor)
DCSConfigEditor: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by /nova/novadaq/releases/R17_00_00/lib/Linux2.6-GCC/libDatabaseUtilsDAQConfig.so)
DCSConfigEditor: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by /nova/novadaq/releases/R17_00_00/lib/Linux2.6-GCC/libDatabaseUtils.so)
DCSConfigEditor: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by /nova/novadaq/releases/R17_00_00/lib/Linux2.6-GCC/libNovaDatabase.so)
DCSConfigEditor: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by /nova/novadaq/releases/R17_00_00/lib/Linux2.6-GCC/libNovaDAQUtilities.so)
DCSConfigEditor: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by /nova/novadaq/releases/R17_00_00/lib/Linux2.6-GCC/libPackageVersion.so)

This is due to a glibc version conflict between SL6 and SL7. Since our dcs machine mounts from test-mgr and uses those releases then the new R17 release built in SL7 doesn’t like the old SL6 glibc version.

Can’t safely update glibc without going to SL7 with the DCS machine which we’re not planning to do quite yet. So the solution it to have the dcs machine set up from R16_00_00 instead of pulling the release from daq-operations.cfg.

For that I’ve added ‘-r R16_00_00’ to the novadaq_setup line in:
novadcs@novadcs-test-01:/home/novadcs/DAQOperationsTools/bin/startDCSConfigEditor.sh

12/01/2020

Detector DAQ [lackey32] DSO/Pedestal Scan

DSO Scan Report Scan after maintenance

8961 12/01/2020 11:17:54

Detector DAQ DSO/Pedestal Scan

Teresa Lackey

Entry Subject: Scan after maintenance

Form:	DSO Scan Report
Detector is cool:	True
Directory name (dcm 1+2):	2020-12-01_11_03_37
Directory name (dcm 3):	2020-12-01_11_08_46
Configuration Name:	TestBeam_AllTrigs_FullDetector
Number of channels for dcms 1 + 2:	2/118/3776
Number of channels for dcm 3:	1/8/256
:	True
Comments?:
Detector is still cool:	True

12/01/2020

Detector DAQ [lackey32] Expert Powering on HV, cooling detector, and starting a run. Going to get a run going for the day so we can make new noise histories for tomorrow with the new APDs/FEBs to see how the swaps went.

8960 12/01/2020 10:50:05

Detector DAQ Expert

Teresa Lackey

Entry Subject: Powering on HV, cooling detector, and starting a run.

Going to get a run going for the day so we can make new noise histories for tomorrow with the new APDs/FEBs to see how the swaps went.

Comments:

Teresa Lackey (lackey32) 12/01/2020 11:01:36

Andrew re-enabled cooling on dcm-5-01-02:42 and dcm-5-01-03:02. Starting up a run now.

Teresa Lackey (lackey32) 12/01/2020 11:02:40

Just kidding. I didn’t do a DSO scan yet. Doing that now…

Teresa Lackey (lackey32) 12/01/2020 11:23:29

Run going.

Teresa Lackey (lackey32) 12/01/2020 11:33:15

Seth is rebooting novadcs-test-0{1,2} to fix the mounts of /nova and /daqlogs. The CSS GUI just disappeared. I’m going to give it a few minutes for things to boot before I restart it.

Teresa Lackey (lackey32) 12/01/2020 11:38:53

Just got a datalogger unresponsive error, probably from me trying to enable some monitoring stuff on mgr-02. I’m trying to move everything there so we can get test-db updated, but perhaps I should just request test-db be updated anyways and go without monitoring for a bit.

Teresa Lackey (lackey32) 12/01/2020 11:41:38

Going to move all monitoring back to test-db so we can try to take a stable run for the day. We’re scheduled for more maintenance tomorrow (if necessary) so I’ll just request test-db be updated then.

Teresa Lackey (lackey32) 12/01/2020 12:00:45

Had to copy bash_profiles/bashrc-novadcs to .bashrc for novadcs. Also restarted the regional IOC. CSS GUI looks good now.

Teresa Lackey (lackey32) 12/01/2020 13:14:39

I changed the ECL link to dbweb8, but we’re still not seeing the Start Run form. Error in logfile (prior to changing to dbweb8) was:
<!DOCTYPE HTML PUBLIC “-//IETF//DTD HTML 2.0//EN”>
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /ECL/novatestbeam/E/xml_post was not found on this server.</p>
</body></html>

Error now is:
Error: [7] – couldn’t connect to host
QObject::connect: Cannot queue arguments of type ‘std::string‘
(Make sure ‘std::string’ is registered using qRegisterMetaType().)

Teresa Lackey (lackey32) 12/01/2020 17:52:46

I’ve been poking at the ganglia webpage on mgr-01 for awhile. Somewhat following https://cdcvs.fnal.gov/redmine/projects/novadaqmonitor/wiki/Instructions_for_compiling_Ganglia_on_novadaq-far-daqmon-2016 just to see which packages were on that list and on test-master, but not on test-mgr-01, I installed ganglia-gmond-python, rrdtool, php-mysql, php-devel, php-gd, and mod_ssl. The webpage is now showing up correctly, and DCM plots are visible. There’s an issue with the gmond_nova.conf files trying to go through eth1 for the manager and farm nodes, but those are controlled by puppet. I’ll add a request to the ticket to update the .conf files to remove the mcast_if line altogether (FD/ND machines don’t have it).

Teresa Lackey (lackey32) 12/01/2020 19:44:26

Stopping run and turning off HV for the night.

11/30/2020

Beamline DAQ [asutton] Expert MWPC WC Tests with new TDCs Dave has replaced the needed TDCs and ASDQ cards. Going to take some runs to check the result of that.

8959 11/30/2020 11:27:17

Beamline DAQ Expert MWPC

Andrew Sutton

Entry Subject: WC Tests with new TDCs

Dave has replaced the needed TDCs and ASDQ cards. Going to take some runs to check the result of that.

Comments:

Andrew Sutton (asutton) 11/30/2020 11:54:57

Also it was just the TDCs swapped not the ASDQs

Andrew Sutton (asutton) 11/30/2020 12:29:39

Just got a run going:

WC3 trigger – run 3583

Andrew Sutton (asutton) 11/30/2020 12:49:55

stopped run 3583

Nearly half of plane 4 (WC 3 x-plane) had no hits at all (not even overflow noise)

The first 16 wires on plane 7 now are showing hits so it’s possible that the TDC channels for those ASDQs were the issue rather than the ASDQs themselves.

Andrew Sutton (asutton) 11/30/2020 12:53:48

WC4 trigger – run 3584

Andrew Sutton (asutton) 12/03/2020 14:14:39

Mike checked all the connections on WC-3 TDC-1. Going to take another run to check it out.

WC3 trigger – run 3585

11/29/2020

Operations [torun]

Shutdown Daily Checklist

8958 11/29/2020 17:04:22

Operations

Yagmur Torun

Entry Subject:

Form:	Shutdown Daily Checklist
Current conditions:
Check cameras:	True
All power in the enclosure and magnet off:	False
Temperature in MC7b:	16.4 C
Dew-point in MC7b:	1.4 C
Dew-point for dry gas output:	-41.1 C
Input pressure for dry-gas flow:	67 psi
Comments:

11/27/2020

Beamline DAQ [lackey32] Expert Linking the TOFPrompt trigger config. I'm going to switch the beamline to running with TOFPrompt and start and stop a beamline run. This should decrease the amount of triggers running through the TDUs, and make them plus OnMon a little happier (OnMon keeps crashing).

8957 11/27/2020 11:00:07

Beamline DAQ Expert

Teresa Lackey

Entry Subject: Linking the TOFPrompt trigger config.

I'm going to switch the beamline to running with TOFPrompt and start and stop a beamline run. This should decrease the amount of triggers running through the TDUs, and make them plus OnMon a little happier (OnMon keeps crashing).

Comments:

Teresa Lackey (lackey32) 11/27/2020 11:03:38: Detector DAQ much happier now. Killing Beamline run.

11/27/2020

Detector DAQ [lackey32] DAQ Software Expert Running the detector for the day. Doing some more work on DAQ stuff today. Most, if not all, of it shouldn't require bringing the run up and down, so I'll leave the detector running as much as possible for stability testing (and so I have data files to run nearline processing on).

8956 11/27/2020 09:58:30

Detector DAQ DAQ Software Expert

Teresa Lackey

Entry Subject: Running the detector for the day.

Doing some more work on DAQ stuff today. Most, if not all, of it shouldn't require bringing the run up and down, so I'll leave the detector running as much as possible for stability testing (and so I have data files to run nearline processing on).

Comments:

Teresa Lackey (lackey32) 11/27/2020 10:03:04

LV and HV on. Cooling detector.

Teresa Lackey (lackey32) 11/27/2020 10:06:26

TDU Control came up blank during reserve resources and TDU Manager stayed yellow in DAQApplicationManager. Releasing and trying again.

Teresa Lackey (lackey32) 11/27/2020 10:11:42

TDU Control came up fine, but TDU-01 thinks it’s 7 am. A sync didn’t help. Going to scrub TDU-01.

Teresa Lackey (lackey32) 11/27/2020 10:26:24

Scrub went fine, but TDUWeb is still unhappy with the time. I suspect the TDU got into some weird state with the high rate of beamline triggers. I’m going to try just starting a run to see what happens. I prefer to avoid rebooting the TDU with no one onsite to recover it if it needs intervention to power cycle.

Teresa Lackey (lackey32) 11/27/2020 10:32:06

Well I see cosmics, so things can’t be too screwed up. We’re not getting the 1Hz, so something is definitely wrong with the spill server side of things. I’ll try restarting the spill server, but having it working isn’t essential today.

Teresa Lackey (lackey32) 11/27/2020 10:38:40

Used the icon, but both NssTDUApps stayed pink. Used the two startBeamSpillBackBoneTB{L}.sh scripts and things are running again. We’re getting 1Hz, and the TDUs are finally happy on the timing status webpage as well. We are getting about 5-7 Hz of beamline triggers, so the high rate is probably the issue.

Teresa Lackey (lackey32) 11/27/2020 10:57:53

Going to move the stuff I installed on mgr-01 for the CR release over to mgr-02. After a day to think about it I’ve decided we’re okay taking up a little bit of space for this on the datadisk, and it’ll save having to mount more things.

Our old datadisk had 459 GB, the new one is 613 GB, and the nusoft area I put on mgr-01 is only 36 GB. I don’t expect nearline-data to get too full, the ND one is only 26 GB. Even with beam doubling the number of root files, I don’t see us taking up too much space, and we can always adjust how quickly we delete files once they’ve been transferred.

I’ve also requested that /nusoft/app and blue3:/nova/data be mounted on mgr-02 so the plotmaking scripts can run on there.

Teresa Lackey (lackey32) 11/27/2020 11:37:20

OnMon viewer is crashing after each subrun, right when it closes the root file. Not sure why, but once I move it over to mgr-02 I’ll keep an eye on it to see if it still does it.

Teresa Lackey (lackey32) 11/27/2020 11:48:01

EVD and OnMon now running on mgr-02. I’m going to work on getting a nearline testrel setup and try to move the processing stuff over to mgr-02. Plotmaking will need to stay on test-db until we have stuff mounted.

Teresa Lackey (lackey32) 11/27/2020 12:21:35

OnMon still crashing at the end of the subrun. It’s behaving like there are two producers running, but I can only see one.

Teresa Lackey (lackey32) 11/27/2020 12:49:22

Just switched nearline processing over to mgr-02 out of a new release S20-11-03. Need to give it a bit to see if everything is working. Going to stop the run and start from a clean slate to see if I can figure out what’s going on with OnMon.

Teresa Lackey (lackey32) 11/27/2020 13:14:26

OnMon made it through the first subrun rollover just fine, so maybe it was a transient issue.

Teresa Lackey (lackey32) 11/27/2020 13:52:51

Killed the VNC servers on test-master.

Teresa Lackey (lackey32) 11/27/2020 15:08:32

I think I’ve fixed the mtu 9000 network check that keeps failing server checks on the SL7 machines, and the mountpoint check for mgr-01. Still need to sort out the ganglia errors.

Teresa Lackey (lackey32) 11/27/2020 15:41:37

Run crashed with DataLogger unresponsive.

Teresa Lackey (lackey32) 11/27/2020 15:45:04

Given the time, rather than starting another run I’m just going to shut things off for the day.

Teresa Lackey (lackey32) 11/27/2020 17:01:50

Changed eth1 to enp2s0f1 in /etc/ganglia/gmond_nova.conf on all farm nodes and restarted gmond_nova on each.

Teresa Lackey (lackey32) 11/30/2020 14:08:14

Looks like the gmond_nova.conf file is controlled by puppet – it reverted back over the weekend. I’ll submit a ticket to get these fixed.

11/25/2020

Detector DAQ [lackey32] DSO/Pedestal Scan

DSO Scan Report First DSO scan since March.

8955 11/25/2020 16:05:10

Detector DAQ DSO/Pedestal Scan

Teresa Lackey

Entry Subject: First DSO scan since March.

Form:	DSO Scan Report
Detector is cool:	True
Directory name (dcm 1+2):	2020-11-25_15_28_28
Directory name (dcm 3):	2020-11-25_15_41_41
Configuration Name:	TestBeam_AllTrigs_FullDetector
Number of channels for dcms 1 + 2:	2/118/3776
Number of channels for dcm 3:	1/8/256
:	True
Comments?:	Scans segfault right at the end, but it's after the reprocessing has happened, so everything still worked.
Detector is still cool:	True

<newer

older>

Version: 8.2