Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......................................X..........] duration: 29.003498202s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:44:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 29.003525444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...X..X.......X....X..............X...X.X.X....X...XXX......] duration: 4.820777463s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:44:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.820803032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..XX...X..XX..X......X.X.X..X..........X...X..X....XXX.X.] duration: 5.674678044s trace: https://jaeger.dev.storj.io/trace/1c86128288c98f1a FAIL (2025-12-29 10:44:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9730 (571h14m44.683342765s) [upload=500816]: Done starting upload Dec 29 10:43:44.812325004 (571h14m44.683408049s) [upload=500816]: Uploading segments... Dec 29 10:43:44.812493375 (571h14m44.68357641s) [upload=500816]: Got next segment. Inline: false Dec 29 10:43:44.812529834 (571h14m44.68361288s) [upload=500816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:43:44.812534123 (571h14m44.683617158s) [upload=500816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:43:45.057692240 (571h14m44.928775285s) [seg_pos={0 0}, upload=500816]: Begin upload segment... Dec 29 10:43:45.057771893 (571h14m44.928854928s) [seg_pos={0 0}, upload=500816]: Done begin upload segment. Dec 29 10:43:45.057857345 (571h14m44.92894039s) [noise=true , address=145.79.227.46:4601 , piece_id=E2XQOSZJLOZJ6DKTG7RN42C7O4B5MSU6ZP7BYRDTPADBIC6T457Q, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.057878997 (571h14m44.928962042s) [noise=true , address=45.91.116.7:4009 , piece_id=EVG24OPQVYCIXG7BIZFD76RRUN6CMGMC377HLALUA2LCY2VSE2DQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.058474190 (571h14m44.929557235s) [noise=true , address=145.79.227.13:4804 , piece_id=IOC4J7TGKDOXINMTMGVVPY5SL53FG7KHGHWKZ5DZYLHIQ76IBURA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.058891334 (571h14m44.929974369s) [noise=true , address=45.91.116.9:4263 , piece_id=M6JKJPNO4PK6DGU3DKBGXOSF6EX3MFMG4TX73P57OO26K2EHX7CA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.059334007 (571h14m44.930417052s) [noise=true , address=145.79.227.50:4374 , piece_id=CQNG7ZSWKC2UVUMK5HCGURYYXTMCYDPHPR6A4LSD2KGFJYNB4OPQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.059795275 (571h14m44.930878321s) [noise=true , address=77.90.31.16:4736 , piece_id=JAN4YVBYFUWTX2WEFTFODEC5U6XO53LS62JT6LVI37BYYSSK3KSA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.060246585 (571h14m44.93132963s) [noise=true , address=145.79.227.2:4114 , piece_id=KV4BIEBUFUMZCINUGICLAX5MQFCUUDSW6IM5Q7TKOQWY7PXDT3SA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.060654791 (571h14m44.931737836s) [noise=true , address=145.79.227.38:4978 , piece_id=QHGH4T7FGAHHPBAURY57FW7T3TYIFLJLJ235DOJW2IM243GNZEVA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.060684448 (571h14m44.931767483s) [noise=true , address=77.90.31.18:4808 , piece_id=ETMZPC62UA4HVVBBWX6C3ED7CYJZJO4UYPLG7YXUCTISP4D4G3NQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.061101021 (571h14m44.932184056s) [noise=true , address=77.90.31.15:4700 , piece_id=XHM6HM6GFTXEURBSMOFFG5JBS4SLBCMWY5232GQGUMOU5FKQLUZA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.061252941 (571h14m44.932335976s) [noise=true , address=145.79.227.14:4838 , piece_id=XX3O2IFLSYRZ6A3HEJKXDZJ6JMPWP7WQHW3CB7EXVB5G5YKL47QQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.061540408 (571h14m44.932623443s) [noise=true , address=145.79.227.4:4187 , piece_id=ACO6O25CHZCLYLCYI4M4WPQJHBWD6QLI7LPNGFZTGBH3FLAXAJ7A, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.062424622 (571h14m44.933507677s) [noise=true , address=145.79.227.48:4303 , piece_id=4QN6GFCEUCQ3BYKCGX26SXL73SY5EAYK3VQ74INSR4E2W7CRIV2Q, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.062920917 (571h14m44.934003952s) [noise=true , address=77.90.31.11:4554 , piece_id=K5QS3JI6RNXMTYQ2HMG7N4AK7ZXR2I3N7IKCM4AD26DRCZT7D74Q, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.063128523 (571h14m44.934211558s) [noise=true , address=145.79.227.6:4258 , piece_id=ID3SDQ5XUURM6TEE4FJO6K2FFLRUWE5G4HLRM7QLGAO7WCJ4DPGA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.063585182 (571h14m44.934668227s) [noise=true , address=45.91.116.4:4940 , piece_id=K3BP6QSCFF7ENKGCXIABDNJPXHRMU6SHWS3QAEKNUVPJVG6KIZ3A, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:46.665424989 (571h14m46.536508024s) [upload=500816]: Next returned nil segment Dec 29 10:43:46.665437012 (571h14m46.536520057s) [upload=500816]: Waiting for error group managing segments... Dec 29 10:43:47.043524879 (571h14m46.914607914s) [noise=true , address=145.79.227.46:4601 , piece_id=E2XQOSZJLOZJ6DKTG7RN42C7O4B5MSU6ZP7BYRDTPADBIC6T457Q, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.045174440 (571h14m46.916257485s) [noise=true , address=145.79.227.6:4258 , piece_id=ID3SDQ5XUURM6TEE4FJO6K2FFLRUWE5G4HLRM7QLGAO7WCJ4DPGA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.047754422 (571h14m46.918837457s) [noise=true , address=145.79.227.14:4838 , piece_id=XX3O2IFLSYRZ6A3HEJKXDZJ6JMPWP7WQHW3CB7EXVB5G5YKL47QQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.049444982 (571h14m46.920528027s) [noise=true , address=145.79.227.38:4978 , piece_id=QHGH4T7FGAHHPBAURY57FW7T3TYIFLJLJ235DOJW2IM243GNZEVA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.050215719 (571h14m46.921298754s) [noise=true , address=145.79.227.50:4374 , piece_id=CQNG7ZSWKC2UVUMK5HCGURYYXTMCYDPHPR6A4LSD2KGFJYNB4OPQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.063331293 (571h14m46.934414338s) [noise=true , address=145.79.227.48:4303 , piece_id=4QN6GFCEUCQ3BYKCGX26SXL73SY5EAYK3VQ74INSR4E2W7CRIV2Q, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.066315125 (571h14m46.93739817s) [noise=true , address=145.79.227.4:4187 , piece_id=ACO6O25CHZCLYLCYI4M4WPQJHBWD6QLI7LPNGFZTGBH3FLAXAJ7A, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.070253274 (571h14m46.941336309s) [noise=true , address=145.79.227.13:4804 , piece_id=IOC4J7TGKDOXINMTMGVVPY5SL53FG7KHGHWKZ5DZYLHIQ76IBURA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.093100769 (571h14m46.964183804s) [noise=true , address=145.79.227.2:4114 , piece_id=KV4BIEBUFUMZCINUGICLAX5MQFCUUDSW6IM5Q7TKOQWY7PXDT3SA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.354385736 (571h14m47.225468781s) [noise=true , address=45.91.116.9:4263 , piece_id=M6JKJPNO4PK6DGU3DKBGXOSF6EX3MFMG4TX73P57OO26K2EHX7CA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.375589781 (571h14m47.246672826s) [noise=true , address=77.90.31.18:4808 , piece_id=ETMZPC62UA4HVVBBWX6C3ED7CYJZJO4UYPLG7YXUCTISP4D4G3NQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.380221450 (571h14m47.251304485s) [noise=true , address=77.90.31.11:4554 , piece_id=K5QS3JI6RNXMTYQ2HMG7N4AK7ZXR2I3N7IKCM4AD26DRCZT7D74Q, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.382004836 (571h14m47.253087881s) [noise=true , address=77.90.31.15:4700 , piece_id=XHM6HM6GFTXEURBSMOFFG5JBS4SLBCMWY5232GQGUMOU5FKQLUZA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.382030896 (571h14m47.253113941s) [seg_pos={0 0}, upload=500816]: Segment reached optimal threshold of 13 pieces. Dec 29 10:43:47.382252988 (571h14m47.253336033s) [noise=true , address=45.91.116.4:4940 , piece_id=K3BP6QSCFF7ENKGCXIABDNJPXHRMU6SHWS3QAEKNUVPJVG6KIZ3A, seg_pos={0 0}, upload=500816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:44348->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:44348->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44348->45.91.116.4:4940: use of closed network connection Dec 29 10:43:47.382323272 (571h14m47.253406317s) [noise=true , address=45.91.116.4:4940 , piece_id=K3BP6QSCFF7ENKGCXIABDNJPXHRMU6SHWS3QAEKNUVPJVG6KIZ3A, seg_pos={0 0}, upload=500816]: Long tail cancellation due to optimal threshold Dec 29 10:43:47.382961558 (571h14m47.254044623s) [noise=true , address=77.90.31.16:4736 , piece_id=JAN4YVBYFUWTX2WEFTFODEC5U6XO53LS62JT6LVI37BYYSSK3KSA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:53394->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 10:43:47.382974823 (571h14m47.254057868s) [noise=true , address=77.90.31.16:4736 , piece_id=JAN4YVBYFUWTX2WEFTFODEC5U6XO53LS62JT6LVI37BYYSSK3KSA, seg_pos={0 0}, upload=500816]: Long tail cancellation due to optimal threshold Dec 29 10:43:47.383139176 (571h14m47.254222221s) [noise=true , address=45.91.116.7:4009 , piece_id=EVG24OPQVYCIXG7BIZFD76RRUN6CMGMC377HLALUA2LCY2VSE2DQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 10:43:47.383148083 (571h14m47.254231118s) [noise=true , address=45.91.116.7:4009 , piece_id=EVG24OPQVYCIXG7BIZFD76RRUN6CMGMC377HLALUA2LCY2VSE2DQ, seg_pos={0 0}, upload=500816]: Long tail cancellation due to optimal threshold Dec 29 10:43:47.383170656 (571h14m47.254253691s) [seg_pos={0 0}, upload=500816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:43:47.383257281 (571h14m47.254340326s) [upload=500816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:43:47.383264665 (571h14m47.2543477s) [upload=500816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:43:47.673393545 (571h14m47.54447659s) [upload=500816]: Done uploading segments... Dec 29 10:43:47.673413082 (571h14m47.544496117s) [upload=500816]: Upload finished. err: <nil> Duration: 5.674705145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..XX...X..XX..X......X.X.X..X..........X...X..X....XXX.X] duration: 30.001806197s trace: https://jaeger.dev.storj.io/trace/1c86128288c98f1a FAIL (2025-12-29 10:44:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9730 (571h14m44.683342765s) [upload=500816]: Done starting upload Dec 29 10:43:44.812325004 (571h14m44.683408049s) [upload=500816]: Uploading segments... Dec 29 10:43:44.812493375 (571h14m44.68357641s) [upload=500816]: Got next segment. Inline: false Dec 29 10:43:44.812529834 (571h14m44.68361288s) [upload=500816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:43:44.812534123 (571h14m44.683617158s) [upload=500816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:43:45.057692240 (571h14m44.928775285s) [seg_pos={0 0}, upload=500816]: Begin upload segment... Dec 29 10:43:45.057771893 (571h14m44.928854928s) [seg_pos={0 0}, upload=500816]: Done begin upload segment. Dec 29 10:43:45.057857345 (571h14m44.92894039s) [noise=true , address=145.79.227.46:4601 , piece_id=E2XQOSZJLOZJ6DKTG7RN42C7O4B5MSU6ZP7BYRDTPADBIC6T457Q, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.057878997 (571h14m44.928962042s) [noise=true , address=45.91.116.7:4009 , piece_id=EVG24OPQVYCIXG7BIZFD76RRUN6CMGMC377HLALUA2LCY2VSE2DQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.058474190 (571h14m44.929557235s) [noise=true , address=145.79.227.13:4804 , piece_id=IOC4J7TGKDOXINMTMGVVPY5SL53FG7KHGHWKZ5DZYLHIQ76IBURA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.058891334 (571h14m44.929974369s) [noise=true , address=45.91.116.9:4263 , piece_id=M6JKJPNO4PK6DGU3DKBGXOSF6EX3MFMG4TX73P57OO26K2EHX7CA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.059334007 (571h14m44.930417052s) [noise=true , address=145.79.227.50:4374 , piece_id=CQNG7ZSWKC2UVUMK5HCGURYYXTMCYDPHPR6A4LSD2KGFJYNB4OPQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.059795275 (571h14m44.930878321s) [noise=true , address=77.90.31.16:4736 , piece_id=JAN4YVBYFUWTX2WEFTFODEC5U6XO53LS62JT6LVI37BYYSSK3KSA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.060246585 (571h14m44.93132963s) [noise=true , address=145.79.227.2:4114 , piece_id=KV4BIEBUFUMZCINUGICLAX5MQFCUUDSW6IM5Q7TKOQWY7PXDT3SA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.060654791 (571h14m44.931737836s) [noise=true , address=145.79.227.38:4978 , piece_id=QHGH4T7FGAHHPBAURY57FW7T3TYIFLJLJ235DOJW2IM243GNZEVA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.060684448 (571h14m44.931767483s) [noise=true , address=77.90.31.18:4808 , piece_id=ETMZPC62UA4HVVBBWX6C3ED7CYJZJO4UYPLG7YXUCTISP4D4G3NQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.061101021 (571h14m44.932184056s) [noise=true , address=77.90.31.15:4700 , piece_id=XHM6HM6GFTXEURBSMOFFG5JBS4SLBCMWY5232GQGUMOU5FKQLUZA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.061252941 (571h14m44.932335976s) [noise=true , address=145.79.227.14:4838 , piece_id=XX3O2IFLSYRZ6A3HEJKXDZJ6JMPWP7WQHW3CB7EXVB5G5YKL47QQ, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.061540408 (571h14m44.932623443s) [noise=true , address=145.79.227.4:4187 , piece_id=ACO6O25CHZCLYLCYI4M4WPQJHBWD6QLI7LPNGFZTGBH3FLAXAJ7A, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.062424622 (571h14m44.933507677s) [noise=true , address=145.79.227.48:4303 , piece_id=4QN6GFCEUCQ3BYKCGX26SXL73SY5EAYK3VQ74INSR4E2W7CRIV2Q, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.062920917 (571h14m44.934003952s) [noise=true , address=77.90.31.11:4554 , piece_id=K5QS3JI6RNXMTYQ2HMG7N4AK7ZXR2I3N7IKCM4AD26DRCZT7D74Q, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.063128523 (571h14m44.934211558s) [noise=true , address=145.79.227.6:4258 , piece_id=ID3SDQ5XUURM6TEE4FJO6K2FFLRUWE5G4HLRM7QLGAO7WCJ4DPGA, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:45.063585182 (571h14m44.934668227s) [noise=true , address=45.91.116.4:4940 , piece_id=K3BP6QSCFF7ENKGCXIABDNJPXHRMU6SHWS3QAEKNUVPJVG6KIZ3A, seg_pos={0 0}, upload=500816]: Uploading piece... Dec 29 10:43:46.665424989 (571h14m46.536508024s) [upload=500816]: Next returned nil segment Dec 29 10:43:46.665437012 (571h14m46.536520057s) [upload=500816]: Waiting for error group managing segments... Dec 29 10:43:47.043524879 (571h14m46.914607914s) [noise=true , address=145.79.227.46:4601 , piece_id=E2XQOSZJLOZJ6DKTG7RN42C7O4B5MSU6ZP7BYRDTPADBIC6T457Q, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.045174440 (571h14m46.916257485s) [noise=true , address=145.79.227.6:4258 , piece_id=ID3SDQ5XUURM6TEE4FJO6K2FFLRUWE5G4HLRM7QLGAO7WCJ4DPGA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.047754422 (571h14m46.918837457s) [noise=true , address=145.79.227.14:4838 , piece_id=XX3O2IFLSYRZ6A3HEJKXDZJ6JMPWP7WQHW3CB7EXVB5G5YKL47QQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.049444982 (571h14m46.920528027s) [noise=true , address=145.79.227.38:4978 , piece_id=QHGH4T7FGAHHPBAURY57FW7T3TYIFLJLJ235DOJW2IM243GNZEVA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.050215719 (571h14m46.921298754s) [noise=true , address=145.79.227.50:4374 , piece_id=CQNG7ZSWKC2UVUMK5HCGURYYXTMCYDPHPR6A4LSD2KGFJYNB4OPQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.063331293 (571h14m46.934414338s) [noise=true , address=145.79.227.48:4303 , piece_id=4QN6GFCEUCQ3BYKCGX26SXL73SY5EAYK3VQ74INSR4E2W7CRIV2Q, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.066315125 (571h14m46.93739817s) [noise=true , address=145.79.227.4:4187 , piece_id=ACO6O25CHZCLYLCYI4M4WPQJHBWD6QLI7LPNGFZTGBH3FLAXAJ7A, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.070253274 (571h14m46.941336309s) [noise=true , address=145.79.227.13:4804 , piece_id=IOC4J7TGKDOXINMTMGVVPY5SL53FG7KHGHWKZ5DZYLHIQ76IBURA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.093100769 (571h14m46.964183804s) [noise=true , address=145.79.227.2:4114 , piece_id=KV4BIEBUFUMZCINUGICLAX5MQFCUUDSW6IM5Q7TKOQWY7PXDT3SA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.354385736 (571h14m47.225468781s) [noise=true , address=45.91.116.9:4263 , piece_id=M6JKJPNO4PK6DGU3DKBGXOSF6EX3MFMG4TX73P57OO26K2EHX7CA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.375589781 (571h14m47.246672826s) [noise=true , address=77.90.31.18:4808 , piece_id=ETMZPC62UA4HVVBBWX6C3ED7CYJZJO4UYPLG7YXUCTISP4D4G3NQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.380221450 (571h14m47.251304485s) [noise=true , address=77.90.31.11:4554 , piece_id=K5QS3JI6RNXMTYQ2HMG7N4AK7ZXR2I3N7IKCM4AD26DRCZT7D74Q, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.382004836 (571h14m47.253087881s) [noise=true , address=77.90.31.15:4700 , piece_id=XHM6HM6GFTXEURBSMOFFG5JBS4SLBCMWY5232GQGUMOU5FKQLUZA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: <nil> Dec 29 10:43:47.382030896 (571h14m47.253113941s) [seg_pos={0 0}, upload=500816]: Segment reached optimal threshold of 13 pieces. Dec 29 10:43:47.382252988 (571h14m47.253336033s) [noise=true , address=45.91.116.4:4940 , piece_id=K3BP6QSCFF7ENKGCXIABDNJPXHRMU6SHWS3QAEKNUVPJVG6KIZ3A, seg_pos={0 0}, upload=500816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:44348->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:44348->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44348->45.91.116.4:4940: use of closed network connection Dec 29 10:43:47.382323272 (571h14m47.253406317s) [noise=true , address=45.91.116.4:4940 , piece_id=K3BP6QSCFF7ENKGCXIABDNJPXHRMU6SHWS3QAEKNUVPJVG6KIZ3A, seg_pos={0 0}, upload=500816]: Long tail cancellation due to optimal threshold Dec 29 10:43:47.382961558 (571h14m47.254044623s) [noise=true , address=77.90.31.16:4736 , piece_id=JAN4YVBYFUWTX2WEFTFODEC5U6XO53LS62JT6LVI37BYYSSK3KSA, seg_pos={0 0}, upload=500816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:53394->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 10:43:47.382974823 (571h14m47.254057868s) [noise=true , address=77.90.31.16:4736 , piece_id=JAN4YVBYFUWTX2WEFTFODEC5U6XO53LS62JT6LVI37BYYSSK3KSA, seg_pos={0 0}, upload=500816]: Long tail cancellation due to optimal threshold Dec 29 10:43:47.383139176 (571h14m47.254222221s) [noise=true , address=45.91.116.7:4009 , piece_id=EVG24OPQVYCIXG7BIZFD76RRUN6CMGMC377HLALUA2LCY2VSE2DQ, seg_pos={0 0}, upload=500816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 10:43:47.383148083 (571h14m47.254231118s) [noise=true , address=45.91.116.7:4009 , piece_id=EVG24OPQVYCIXG7BIZFD76RRUN6CMGMC377HLALUA2LCY2VSE2DQ, seg_pos={0 0}, upload=500816]: Long tail cancellation due to optimal threshold Dec 29 10:43:47.383170656 (571h14m47.254253691s) [seg_pos={0 0}, upload=500816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:43:47.383257281 (571h14m47.254340326s) [upload=500816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:43:47.383264665 (571h14m47.2543477s) [upload=500816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:43:47.673393545 (571h14m47.54447659s) [upload=500816]: Done uploading segments... Dec 29 10:43:47.673413082 (571h14m47.544496117s) [upload=500816]: Upload finished. err: <nil> Duration: 30.001827718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................................X.........] duration: 31.373601502s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:44:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 31.373631109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....X..X.......X....X..............X...X.X.X....X...XXX.....] duration: 3.465523284s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:43:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465548823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X..XX...X..XX..X......X.X.X..X..........X...X..X....XXX.] duration: 6.244725617s trace: https://jaeger.dev.storj.io/trace/5042cbf96292ad5d FAIL (2025-12-29 10:43:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: VSS4XXZTUJODMTKZUDYF6ZKXERCCP4JJTP5EF5GDLBAYVZ6QR5TQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SIH5MF7U7DPJ4ZGPON2LXTCPJALQT457J7JG6YZ4MIV5F77M24TA) context deadline exceeded ======LOGS====== 29 10:42:53.528028071 (571h13m53.399111116s) [upload=500802]: Starting upload Dec 29 10:42:53.528117270 (571h13m53.399200315s) [upload=500802]: Done starting upload Dec 29 10:42:53.528250854 (571h13m53.399333909s) [upload=500802]: Uploading segments... Dec 29 10:42:53.528512423 (571h13m53.399595468s) [upload=500802]: Got next segment. Inline: false Dec 29 10:42:53.528559923 (571h13m53.399642978s) [upload=500802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:42:53.528566996 (571h13m53.399650041s) [upload=500802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:42:53.771945507 (571h13m53.643028562s) [seg_pos={0 0}, upload=500802]: Begin upload segment... Dec 29 10:42:53.771996754 (571h13m53.643079789s) [seg_pos={0 0}, upload=500802]: Done begin upload segment. Dec 29 10:42:53.772024968 (571h13m53.643108003s) [noise=true , address=145.79.227.46:4603 , piece_id=DWUK5OJXC3FMNMWHNTPDRBKPBE75HBXB7OGZVEXJBEKTCV7QMGJA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.772134306 (571h13m53.643217351s) [noise=true , address=145.79.227.42:4121 , piece_id=LXUA463BS6KNK4A6JK3X5XZNY7FICDMXDY7Z25GZCOTPP2HJVN6Q, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.772725962 (571h13m53.643808997s) [noise=true , address=145.79.227.10:4401 , piece_id=UEZHTUCWZCRBESLIHAF4HI653F6XKFVQLSXYRAS4E43MKWBN7KQA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.773041032 (571h13m53.644124067s) [noise=true , address=145.79.227.16:4909 , piece_id=XIHE3Z2FAAU2YA367BZW5ECZFFUWE6LEIO3VO2TUUTJOYCTBG77Q, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.773456463 (571h13m53.644539498s) [noise=true , address=145.79.227.6:4258 , piece_id=OFMBB3HYIB5GQG4L4AEGZKUSSLX5F3ZPFGRSGQJE6TENIT2X3TAA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774088416 (571h13m53.645171451s) [noise=true , address=145.79.227.50:4375 , piece_id=Z2YXTM5MWBKAYJFUPXA6IOPUJ56O7EWEW5HI4W7PN6NZXSTNBHZA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774154082 (571h13m53.645237117s) [noise=true , address=45.91.116.11:4117 , piece_id=LQ65LVE7N6GIBDN7MLXBJYL44YZ3KLQRPEZV4YIAZN3T3757WI4A, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774506753 (571h13m53.645589788s) [noise=true , address=45.91.116.8:4045 , piece_id=RDC2Y56KKRZIR6BZT2JDFNPQVSQDX2KQ77NQY3IMOUNBZCJMF5UA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774594329 (571h13m53.645677374s) [noise=true , address=145.79.227.18:4006 , piece_id=NCEU2DRJLXGO4DF6TRMXBUTMEUQRWDZP37ABTHVCIFHNVK5IYSKQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774813087 (571h13m53.645896132s) [noise=true , address=145.79.227.4:4186 , piece_id=QO6ELYHVW7CVLGL6LVGJM44USJIHY55KUCO3LTTMURYPM6JSVCFA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775109701 (571h13m53.646192736s) [noise=true , address=77.90.31.11:4554 , piece_id=OI3PBU2Z6B6AVR5IJEGRXSOI7CUUX2ZGNFSG4KPAEZKACOVU5YHA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775195895 (571h13m53.646278941s) [noise=true , address=77.90.31.12:4589 , piece_id=VSS4XXZTUJODMTKZUDYF6ZKXERCCP4JJTP5EF5GDLBAYVZ6QR5TQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775543077 (571h13m53.646626112s) [noise=true , address=77.90.31.18:4805 , piece_id=VHP3ZZXUAPAJREG4ILOAPLGUKGLRNPN3EQQ6RCHDBZKJ4WBK7XXQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775660710 (571h13m53.646743755s) [noise=true , address=77.90.31.17:4773 , piece_id=ZESZUIYIHAZGLH6XVMB5F4SE6JETSVA6D7DU5VQZHMUT5J4RU22Q, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775870399 (571h13m53.646953434s) [noise=true , address=145.79.227.3:4149 , piece_id=I7VAKYWSRYPRFDTJXQHU76CDJV3BIZE2OIY75KXWQ2Q5D4DYGJDQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.776046294 (571h13m53.647129339s) [noise=true , address=77.90.31.19:4844 , piece_id=SIH5MF7U7DPJ4ZGPON2LXTCPJALQT457J7JG6YZ4MIV5F77M24TA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:55.471186336 (571h13m55.342269381s) [upload=500802]: Next returned nil segment Dec 29 10:42:55.471213277 (571h13m55.342296312s) [upload=500802]: Waiting for error group managing segments... Dec 29 10:42:55.752414718 (571h13m55.623497753s) [noise=true , address=145.79.227.16:4909 , piece_id=XIHE3Z2FAAU2YA367BZW5ECZFFUWE6LEIO3VO2TUUTJOYCTBG77Q, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.757448954 (571h13m55.628531989s) [noise=true , address=145.79.227.42:4121 , piece_id=LXUA463BS6KNK4A6JK3X5XZNY7FICDMXDY7Z25GZCOTPP2HJVN6Q, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.760071597 (571h13m55.631154633s) [noise=true , address=145.79.227.3:4149 , piece_id=I7VAKYWSRYPRFDTJXQHU76CDJV3BIZE2OIY75KXWQ2Q5D4DYGJDQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.760555188 (571h13m55.631638223s) [noise=true , address=145.79.227.50:4375 , piece_id=Z2YXTM5MWBKAYJFUPXA6IOPUJ56O7EWEW5HI4W7PN6NZXSTNBHZA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.762011522 (571h13m55.633094557s) [noise=true , address=145.79.227.46:4603 , piece_id=DWUK5OJXC3FMNMWHNTPDRBKPBE75HBXB7OGZVEXJBEKTCV7QMGJA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.762130949 (571h13m55.633213984s) [noise=true , address=145.79.227.18:4006 , piece_id=NCEU2DRJLXGO4DF6TRMXBUTMEUQRWDZP37ABTHVCIFHNVK5IYSKQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.762380253 (571h13m55.633463288s) [noise=true , address=145.79.227.10:4401 , piece_id=UEZHTUCWZCRBESLIHAF4HI653F6XKFVQLSXYRAS4E43MKWBN7KQA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.783095619 (571h13m55.654178664s) [noise=true , address=145.79.227.6:4258 , piece_id=OFMBB3HYIB5GQG4L4AEGZKUSSLX5F3ZPFGRSGQJE6TENIT2X3TAA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.783334744 (571h13m55.654417789s) [noise=true , address=145.79.227.4:4186 , piece_id=QO6ELYHVW7CVLGL6LVGJM44USJIHY55KUCO3LTTMURYPM6JSVCFA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.995028481 (571h13m55.866111526s) [noise=true , address=77.90.31.12:4589 , piece_id=VSS4XXZTUJODMTKZUDYF6ZKXERCCP4JJTP5EF5GDLBAYVZ6QR5TQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.042561441 (571h13m55.913644486s) [noise=true , address=77.90.31.17:4773 , piece_id=ZESZUIYIHAZGLH6XVMB5F4SE6JETSVA6D7DU5VQZHMUT5J4RU22Q, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.076819264 (571h13m55.947902299s) [noise=true , address=77.90.31.11:4554 , piece_id=OI3PBU2Z6B6AVR5IJEGRXSOI7CUUX2ZGNFSG4KPAEZKACOVU5YHA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.076893284 (571h13m55.947976329s) [noise=true , address=77.90.31.19:4844 , piece_id=SIH5MF7U7DPJ4ZGPON2LXTCPJALQT457J7JG6YZ4MIV5F77M24TA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.076914043 (571h13m55.947997078s) [seg_pos={0 0}, upload=500802]: Segment reached optimal threshold of 13 pieces. Dec 29 10:42:56.077130025 (571h13m55.94821307s) [noise=true , address=45.91.116.11:4117 , piece_id=LQ65LVE7N6GIBDN7MLXBJYL44YZ3KLQRPEZV4YIAZN3T3757WI4A, seg_pos={0 0}, upload=500802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 10:42:56.077136708 (571h13m55.948219763s) [noise=true , address=45.91.116.11:4117 , piece_id=LQ65LVE7N6GIBDN7MLXBJYL44YZ3KLQRPEZV4YIAZN3T3757WI4A, seg_pos={0 0}, upload=500802]: Long tail cancellation due to optimal threshold Dec 29 10:42:56.077148730 (571h13m55.948231766s) [noise=true , address=77.90.31.18:4805 , piece_id=VHP3ZZXUAPAJREG4ILOAPLGUKGLRNPN3EQQ6RCHDBZKJ4WBK7XXQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 10:42:56.077156305 (571h13m55.94823934s) [noise=true , address=77.90.31.18:4805 , piece_id=VHP3ZZXUAPAJREG4ILOAPLGUKGLRNPN3EQQ6RCHDBZKJ4WBK7XXQ, seg_pos={0 0}, upload=500802]: Long tail cancellation due to optimal threshold Dec 29 10:42:56.077293296 (571h13m55.948376341s) [noise=true , address=45.91.116.8:4045 , piece_id=RDC2Y56KKRZIR6BZT2JDFNPQVSQDX2KQ77NQY3IMOUNBZCJMF5UA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:34806->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:34806->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34806->45.91.116.8:4045: use of closed network connection Dec 29 10:42:56.077300149 (571h13m55.948383195s) [noise=true , address=45.91.116.8:4045 , piece_id=RDC2Y56KKRZIR6BZT2JDFNPQVSQDX2KQ77NQY3IMOUNBZCJMF5UA, seg_pos={0 0}, upload=500802]: Long tail cancellation due to optimal threshold Dec 29 10:42:56.077316771 (571h13m55.948399806s) [seg_pos={0 0}, upload=500802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:42:56.077404158 (571h13m55.948487193s) [upload=500802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:42:56.077408916 (571h13m55.948491962s) [upload=500802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:42:56.362608288 (571h13m56.233691323s) [upload=500802]: Done uploading segments... Dec 29 10:42:56.362632455 (571h13m56.2337155s) [upload=500802]: Upload finished. err: <nil> Duration: 6.244744483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X...X..XX...X..XX..X......X.X.X..X..........X...X..X....XXX] duration: 30.002826045s trace: https://jaeger.dev.storj.io/trace/5042cbf96292ad5d FAIL (2025-12-29 10:43:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: VSS4XXZTUJODMTKZUDYF6ZKXERCCP4JJTP5EF5GDLBAYVZ6QR5TQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SIH5MF7U7DPJ4ZGPON2LXTCPJALQT457J7JG6YZ4MIV5F77M24TA) context deadline exceeded ======LOGS====== 29 10:42:53.528028071 (571h13m53.399111116s) [upload=500802]: Starting upload Dec 29 10:42:53.528117270 (571h13m53.399200315s) [upload=500802]: Done starting upload Dec 29 10:42:53.528250854 (571h13m53.399333909s) [upload=500802]: Uploading segments... Dec 29 10:42:53.528512423 (571h13m53.399595468s) [upload=500802]: Got next segment. Inline: false Dec 29 10:42:53.528559923 (571h13m53.399642978s) [upload=500802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:42:53.528566996 (571h13m53.399650041s) [upload=500802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:42:53.771945507 (571h13m53.643028562s) [seg_pos={0 0}, upload=500802]: Begin upload segment... Dec 29 10:42:53.771996754 (571h13m53.643079789s) [seg_pos={0 0}, upload=500802]: Done begin upload segment. Dec 29 10:42:53.772024968 (571h13m53.643108003s) [noise=true , address=145.79.227.46:4603 , piece_id=DWUK5OJXC3FMNMWHNTPDRBKPBE75HBXB7OGZVEXJBEKTCV7QMGJA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.772134306 (571h13m53.643217351s) [noise=true , address=145.79.227.42:4121 , piece_id=LXUA463BS6KNK4A6JK3X5XZNY7FICDMXDY7Z25GZCOTPP2HJVN6Q, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.772725962 (571h13m53.643808997s) [noise=true , address=145.79.227.10:4401 , piece_id=UEZHTUCWZCRBESLIHAF4HI653F6XKFVQLSXYRAS4E43MKWBN7KQA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.773041032 (571h13m53.644124067s) [noise=true , address=145.79.227.16:4909 , piece_id=XIHE3Z2FAAU2YA367BZW5ECZFFUWE6LEIO3VO2TUUTJOYCTBG77Q, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.773456463 (571h13m53.644539498s) [noise=true , address=145.79.227.6:4258 , piece_id=OFMBB3HYIB5GQG4L4AEGZKUSSLX5F3ZPFGRSGQJE6TENIT2X3TAA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774088416 (571h13m53.645171451s) [noise=true , address=145.79.227.50:4375 , piece_id=Z2YXTM5MWBKAYJFUPXA6IOPUJ56O7EWEW5HI4W7PN6NZXSTNBHZA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774154082 (571h13m53.645237117s) [noise=true , address=45.91.116.11:4117 , piece_id=LQ65LVE7N6GIBDN7MLXBJYL44YZ3KLQRPEZV4YIAZN3T3757WI4A, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774506753 (571h13m53.645589788s) [noise=true , address=45.91.116.8:4045 , piece_id=RDC2Y56KKRZIR6BZT2JDFNPQVSQDX2KQ77NQY3IMOUNBZCJMF5UA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774594329 (571h13m53.645677374s) [noise=true , address=145.79.227.18:4006 , piece_id=NCEU2DRJLXGO4DF6TRMXBUTMEUQRWDZP37ABTHVCIFHNVK5IYSKQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.774813087 (571h13m53.645896132s) [noise=true , address=145.79.227.4:4186 , piece_id=QO6ELYHVW7CVLGL6LVGJM44USJIHY55KUCO3LTTMURYPM6JSVCFA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775109701 (571h13m53.646192736s) [noise=true , address=77.90.31.11:4554 , piece_id=OI3PBU2Z6B6AVR5IJEGRXSOI7CUUX2ZGNFSG4KPAEZKACOVU5YHA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775195895 (571h13m53.646278941s) [noise=true , address=77.90.31.12:4589 , piece_id=VSS4XXZTUJODMTKZUDYF6ZKXERCCP4JJTP5EF5GDLBAYVZ6QR5TQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775543077 (571h13m53.646626112s) [noise=true , address=77.90.31.18:4805 , piece_id=VHP3ZZXUAPAJREG4ILOAPLGUKGLRNPN3EQQ6RCHDBZKJ4WBK7XXQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775660710 (571h13m53.646743755s) [noise=true , address=77.90.31.17:4773 , piece_id=ZESZUIYIHAZGLH6XVMB5F4SE6JETSVA6D7DU5VQZHMUT5J4RU22Q, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.775870399 (571h13m53.646953434s) [noise=true , address=145.79.227.3:4149 , piece_id=I7VAKYWSRYPRFDTJXQHU76CDJV3BIZE2OIY75KXWQ2Q5D4DYGJDQ, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:53.776046294 (571h13m53.647129339s) [noise=true , address=77.90.31.19:4844 , piece_id=SIH5MF7U7DPJ4ZGPON2LXTCPJALQT457J7JG6YZ4MIV5F77M24TA, seg_pos={0 0}, upload=500802]: Uploading piece... Dec 29 10:42:55.471186336 (571h13m55.342269381s) [upload=500802]: Next returned nil segment Dec 29 10:42:55.471213277 (571h13m55.342296312s) [upload=500802]: Waiting for error group managing segments... Dec 29 10:42:55.752414718 (571h13m55.623497753s) [noise=true , address=145.79.227.16:4909 , piece_id=XIHE3Z2FAAU2YA367BZW5ECZFFUWE6LEIO3VO2TUUTJOYCTBG77Q, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.757448954 (571h13m55.628531989s) [noise=true , address=145.79.227.42:4121 , piece_id=LXUA463BS6KNK4A6JK3X5XZNY7FICDMXDY7Z25GZCOTPP2HJVN6Q, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.760071597 (571h13m55.631154633s) [noise=true , address=145.79.227.3:4149 , piece_id=I7VAKYWSRYPRFDTJXQHU76CDJV3BIZE2OIY75KXWQ2Q5D4DYGJDQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.760555188 (571h13m55.631638223s) [noise=true , address=145.79.227.50:4375 , piece_id=Z2YXTM5MWBKAYJFUPXA6IOPUJ56O7EWEW5HI4W7PN6NZXSTNBHZA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.762011522 (571h13m55.633094557s) [noise=true , address=145.79.227.46:4603 , piece_id=DWUK5OJXC3FMNMWHNTPDRBKPBE75HBXB7OGZVEXJBEKTCV7QMGJA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.762130949 (571h13m55.633213984s) [noise=true , address=145.79.227.18:4006 , piece_id=NCEU2DRJLXGO4DF6TRMXBUTMEUQRWDZP37ABTHVCIFHNVK5IYSKQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.762380253 (571h13m55.633463288s) [noise=true , address=145.79.227.10:4401 , piece_id=UEZHTUCWZCRBESLIHAF4HI653F6XKFVQLSXYRAS4E43MKWBN7KQA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.783095619 (571h13m55.654178664s) [noise=true , address=145.79.227.6:4258 , piece_id=OFMBB3HYIB5GQG4L4AEGZKUSSLX5F3ZPFGRSGQJE6TENIT2X3TAA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.783334744 (571h13m55.654417789s) [noise=true , address=145.79.227.4:4186 , piece_id=QO6ELYHVW7CVLGL6LVGJM44USJIHY55KUCO3LTTMURYPM6JSVCFA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:55.995028481 (571h13m55.866111526s) [noise=true , address=77.90.31.12:4589 , piece_id=VSS4XXZTUJODMTKZUDYF6ZKXERCCP4JJTP5EF5GDLBAYVZ6QR5TQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.042561441 (571h13m55.913644486s) [noise=true , address=77.90.31.17:4773 , piece_id=ZESZUIYIHAZGLH6XVMB5F4SE6JETSVA6D7DU5VQZHMUT5J4RU22Q, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.076819264 (571h13m55.947902299s) [noise=true , address=77.90.31.11:4554 , piece_id=OI3PBU2Z6B6AVR5IJEGRXSOI7CUUX2ZGNFSG4KPAEZKACOVU5YHA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.076893284 (571h13m55.947976329s) [noise=true , address=77.90.31.19:4844 , piece_id=SIH5MF7U7DPJ4ZGPON2LXTCPJALQT457J7JG6YZ4MIV5F77M24TA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: <nil> Dec 29 10:42:56.076914043 (571h13m55.947997078s) [seg_pos={0 0}, upload=500802]: Segment reached optimal threshold of 13 pieces. Dec 29 10:42:56.077130025 (571h13m55.94821307s) [noise=true , address=45.91.116.11:4117 , piece_id=LQ65LVE7N6GIBDN7MLXBJYL44YZ3KLQRPEZV4YIAZN3T3757WI4A, seg_pos={0 0}, upload=500802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 10:42:56.077136708 (571h13m55.948219763s) [noise=true , address=45.91.116.11:4117 , piece_id=LQ65LVE7N6GIBDN7MLXBJYL44YZ3KLQRPEZV4YIAZN3T3757WI4A, seg_pos={0 0}, upload=500802]: Long tail cancellation due to optimal threshold Dec 29 10:42:56.077148730 (571h13m55.948231766s) [noise=true , address=77.90.31.18:4805 , piece_id=VHP3ZZXUAPAJREG4ILOAPLGUKGLRNPN3EQQ6RCHDBZKJ4WBK7XXQ, seg_pos={0 0}, upload=500802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 10:42:56.077156305 (571h13m55.94823934s) [noise=true , address=77.90.31.18:4805 , piece_id=VHP3ZZXUAPAJREG4ILOAPLGUKGLRNPN3EQQ6RCHDBZKJ4WBK7XXQ, seg_pos={0 0}, upload=500802]: Long tail cancellation due to optimal threshold Dec 29 10:42:56.077293296 (571h13m55.948376341s) [noise=true , address=45.91.116.8:4045 , piece_id=RDC2Y56KKRZIR6BZT2JDFNPQVSQDX2KQ77NQY3IMOUNBZCJMF5UA, seg_pos={0 0}, upload=500802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:34806->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:34806->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34806->45.91.116.8:4045: use of closed network connection Dec 29 10:42:56.077300149 (571h13m55.948383195s) [noise=true , address=45.91.116.8:4045 , piece_id=RDC2Y56KKRZIR6BZT2JDFNPQVSQDX2KQ77NQY3IMOUNBZCJMF5UA, seg_pos={0 0}, upload=500802]: Long tail cancellation due to optimal threshold Dec 29 10:42:56.077316771 (571h13m55.948399806s) [seg_pos={0 0}, upload=500802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:42:56.077404158 (571h13m55.948487193s) [upload=500802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:42:56.077408916 (571h13m55.948491962s) [upload=500802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:42:56.362608288 (571h13m56.233691323s) [upload=500802]: Done uploading segments... Dec 29 10:42:56.362632455 (571h13m56.2337155s) [upload=500802]: Upload finished. err: <nil> Duration: 30.002973555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................................X........] duration: 30.893903731s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:43:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 30.89392974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....X..X.......X....X..............X...X.X.X....X...XXX....] duration: 3.428717757s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:43:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428744167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X...X..XX...X..XX..X......X.X.X..X..........X...X..X....XX] duration: 30.003194051s trace: https://jaeger.dev.storj.io/trace/44565a835fc59005 FAIL (2025-12-29 10:42:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: BGYSBUYBDBTS2LTWBXDZP6ZYA5EQFG6KD4ZUQ4KJ6YNAMIFENJZA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 7MFWJWFYS5BNZGZQDAQXPFORURDU3MIBOMRVLAAIQPBLIOVNM2RA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ZRCHHUJR5BNZP22HVEGBYUMBSOHE7Y7HY2XLPZG3DHJKBBALVRNA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: USFZDEGWWZJBYKFLF5I4TF24NUZZ62ABHKMD4FPOQGDMJPWS2SPQ) context deadline exceeded ======LOGS====== Dec 29 10:42:06.150147158 (571h13m6.021230193s) [upload=500790]: Starting upload Dec 29 10:42:06.150234154 (571h13m6.021317199s) [upload=500790]: Done starting upload Dec 29 10:42:06.150280732 (571h13m6.021363777s) [upload=500790]: Uploading segments... Dec 29 10:42:06.150476025 (571h13m6.02155907s) [upload=500790]: Got next segment. Inline: false Dec 29 10:42:06.150518195 (571h13m6.02160124s) [upload=500790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:42:06.150523975 (571h13m6.021607021s) [upload=500790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:42:06.394633718 (571h13m6.265716763s) [seg_pos={0 0}, upload=500790]: Begin upload segment... Dec 29 10:42:06.394708851 (571h13m6.265791886s) [seg_pos={0 0}, upload=500790]: Done begin upload segment. Dec 29 10:42:06.394744659 (571h13m6.265827704s) [noise=true , address=145.79.227.3:4149 , piece_id=TTEQVIR3JCV4XEIO3FV43MYDVJOSGVQA57J7XVI4UCVQJ3I62V2Q, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.394845351 (571h13m6.265928386s) [noise=true , address=45.91.116.8:4048 , piece_id=S3ZTLZO3EVKC7PCLOH66ZW5R2DDVE6TNR2Q7ZW3X3T6P575T2RQQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.395629654 (571h13m6.266712699s) [noise=true , address=145.79.227.50:4375 , piece_id=36KODVDUB7BR6LU3OUMZESGA7MN22ULC4P5Q3H3SXQUZKODC72FA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.395792154 (571h13m6.266875199s) [noise=true , address=45.91.116.11:4117 , piece_id=NBJBN6CXW7SQWQHH4IWGPRMGWFWLQH5PIJQDPUN6KCAY4OOCSQVQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.396112143 (571h13m6.267195188s) [noise=true , address=45.91.116.7:4011 , piece_id=7MFWJWFYS5BNZGZQDAQXPFORURDU3MIBOMRVLAAIQPBLIOVNM2RA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.396231540 (571h13m6.267314585s) [noise=true , address=45.91.116.9:4262 , piece_id=ZRCHHUJR5BNZP22HVEGBYUMBSOHE7Y7HY2XLPZG3DHJKBBALVRNA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.396714790 (571h13m6.267797825s) [noise=true , address=77.90.31.16:4736 , piece_id=USFZDEGWWZJBYKFLF5I4TF24NUZZ62ABHKMD4FPOQGDMJPWS2SPQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.396723517 (571h13m6.267806552s) [noise=true , address=45.91.116.5:4300 , piece_id=B2KAULERQHXZVDIJFQM3MQY7N7F7NTVHT2YEIBYJCGXYKMSNS5JQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.397141062 (571h13m6.268224098s) [noise=true , address=145.79.227.5:4223 , piece_id=ICCGNYNJMIO5HU7TDLR3A65TNN66U6XFKUTPC2Y4JOREHJ7Y2ZZQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.397177512 (571h13m6.268260557s) [noise=true , address=45.91.116.10:4083 , piece_id=CNURYMARMWWPE3ZHH3HGEGZ4LDT5WKS5JLYGQLJLXWEXCJSSJV5Q, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.397766103 (571h13m6.268849148s) [noise=true , address=77.90.31.13:4627 , piece_id=J4E3FMCL7Z6ZIYI4Q3RATPHJYOGBQFILUPS556QEHB3MPKKWGMUA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.399571900 (571h13m6.270654945s) [noise=true , address=45.91.116.12:4226 , piece_id=ISA4M367CQ6J2D3KYZDNCT624BFNIXQYS326KPRPXRKVILYHKERQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.400860514 (571h13m6.271943559s) [noise=true , address=77.90.31.12:4590 , piece_id=BGYSBUYBDBTS2LTWBXDZP6ZYA5EQFG6KD4ZUQ4KJ6YNAMIFENJZA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.401827566 (571h13m6.272910621s) [noise=true , address=77.90.31.19:4844 , piece_id=DJ2BJBDKQVT6QKVJB4L6S6UJB6YKBE47M56QBPMQIEMVEFRNEKOA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.403231169 (571h13m6.274314214s) [noise=true , address=145.79.227.13:4803 , piece_id=JJPKRYILJXBQMQ6SHSNHGNMI7FZATTLA5YKKJCK6RL4JT6YSLUWQ, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:06.408309118 (571h13m6.279392163s) [noise=true , address=145.79.227.19:4078 , piece_id=JDYAXPN5B7RKG6S5G2ISUR3HOM4HWJBKTTIX6ZMRU2R2UESOMVJA, seg_pos={0 0}, upload=500790]: Uploading piece... Dec 29 10:42:08.092008247 (571h13m7.963091282s) [upload=500790]: Next returned nil segment Dec 29 10:42:08.092035119 (571h13m7.963118164s) [upload=500790]: Waiting for error group managing segments... Dec 29 10:42:08.384894599 (571h13m8.255977644s) [noise=true , address=145.79.227.5:4223 , piece_id=ICCGNYNJMIO5HU7TDLR3A65TNN66U6XFKUTPC2Y4JOREHJ7Y2ZZQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.405778284 (571h13m8.276861319s) [noise=true , address=145.79.227.3:4149 , piece_id=TTEQVIR3JCV4XEIO3FV43MYDVJOSGVQA57J7XVI4UCVQJ3I62V2Q, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.405912910 (571h13m8.276995955s) [noise=true , address=145.79.227.19:4078 , piece_id=JDYAXPN5B7RKG6S5G2ISUR3HOM4HWJBKTTIX6ZMRU2R2UESOMVJA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.419788401 (571h13m8.290871446s) [noise=true , address=145.79.227.13:4803 , piece_id=JJPKRYILJXBQMQ6SHSNHGNMI7FZATTLA5YKKJCK6RL4JT6YSLUWQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.423952250 (571h13m8.295035295s) [noise=true , address=145.79.227.50:4375 , piece_id=36KODVDUB7BR6LU3OUMZESGA7MN22ULC4P5Q3H3SXQUZKODC72FA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.680791181 (571h13m8.551874216s) [noise=true , address=77.90.31.19:4844 , piece_id=DJ2BJBDKQVT6QKVJB4L6S6UJB6YKBE47M56QBPMQIEMVEFRNEKOA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.680997763 (571h13m8.552080808s) [noise=true , address=45.91.116.5:4300 , piece_id=B2KAULERQHXZVDIJFQM3MQY7N7F7NTVHT2YEIBYJCGXYKMSNS5JQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.690139391 (571h13m8.561222426s) [noise=true , address=77.90.31.16:4736 , piece_id=USFZDEGWWZJBYKFLF5I4TF24NUZZ62ABHKMD4FPOQGDMJPWS2SPQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.693092043 (571h13m8.564175088s) [noise=true , address=77.90.31.12:4590 , piece_id=BGYSBUYBDBTS2LTWBXDZP6ZYA5EQFG6KD4ZUQ4KJ6YNAMIFENJZA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.726642769 (571h13m8.597725814s) [noise=true , address=45.91.116.12:4226 , piece_id=ISA4M367CQ6J2D3KYZDNCT624BFNIXQYS326KPRPXRKVILYHKERQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.743685120 (571h13m8.614769377s) [noise=true , address=45.91.116.9:4262 , piece_id=ZRCHHUJR5BNZP22HVEGBYUMBSOHE7Y7HY2XLPZG3DHJKBBALVRNA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.750600869 (571h13m8.621683914s) [noise=true , address=45.91.116.7:4011 , piece_id=7MFWJWFYS5BNZGZQDAQXPFORURDU3MIBOMRVLAAIQPBLIOVNM2RA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.792052379 (571h13m8.663135424s) [noise=true , address=45.91.116.11:4117 , piece_id=NBJBN6CXW7SQWQHH4IWGPRMGWFWLQH5PIJQDPUN6KCAY4OOCSQVQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: <nil> Dec 29 10:42:08.792078769 (571h13m8.663161814s) [seg_pos={0 0}, upload=500790]: Segment reached optimal threshold of 13 pieces. Dec 29 10:42:08.792340938 (571h13m8.663423973s) [noise=true , address=45.91.116.10:4083 , piece_id=CNURYMARMWWPE3ZHH3HGEGZ4LDT5WKS5JLYGQLJLXWEXCJSSJV5Q, seg_pos={0 0}, upload=500790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:37226->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:37226->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37226->45.91.116.10:4083: use of closed network connection Dec 29 10:42:08.792351288 (571h13m8.663434333s) [noise=true , address=45.91.116.10:4083 , piece_id=CNURYMARMWWPE3ZHH3HGEGZ4LDT5WKS5JLYGQLJLXWEXCJSSJV5Q, seg_pos={0 0}, upload=500790]: Long tail cancellation due to optimal threshold Dec 29 10:42:08.792718447 (571h13m8.663801492s) [noise=true , address=77.90.31.13:4627 , piece_id=J4E3FMCL7Z6ZIYI4Q3RATPHJYOGBQFILUPS556QEHB3MPKKWGMUA, seg_pos={0 0}, upload=500790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 10:42:08.792728576 (571h13m8.663811611s) [noise=true , address=77.90.31.13:4627 , piece_id=J4E3FMCL7Z6ZIYI4Q3RATPHJYOGBQFILUPS556QEHB3MPKKWGMUA, seg_pos={0 0}, upload=500790]: Long tail cancellation due to optimal threshold Dec 29 10:42:08.792914089 (571h13m8.663997144s) [noise=true , address=45.91.116.8:4048 , piece_id=S3ZTLZO3EVKC7PCLOH66ZW5R2DDVE6TNR2Q7ZW3X3T6P575T2RQQ, seg_pos={0 0}, upload=500790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 10:42:08.792924760 (571h13m8.664007805s) [noise=true , address=45.91.116.8:4048 , piece_id=S3ZTLZO3EVKC7PCLOH66ZW5R2DDVE6TNR2Q7ZW3X3T6P575T2RQQ, seg_pos={0 0}, upload=500790]: Long tail cancellation due to optimal threshold Dec 29 10:42:08.792947473 (571h13m8.664030518s) [seg_pos={0 0}, upload=500790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:42:08.793031503 (571h13m8.664114548s) [upload=500790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:42:08.793037835 (571h13m8.66412088s) [upload=500790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:42:09.079300343 (571h13m8.950383378s) [upload=500790]: Done uploading segments... Dec 29 10:42:09.079321262 (571h13m8.950404297s) [upload=500790]: Upload finished. err: <nil> Duration: 30.003221414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................................X.......] duration: 29.985281236s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 29.985294741s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......X..X.......X....X..............X...X.X.X....X...XXX...] duration: 3.66117675s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:42:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661204593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X...X..XX...X..XX..X......X.X.X..X..........X...X..X....X] duration: 30.003979988s trace: https://jaeger.dev.storj.io/trace/2e9dafd0c1c544cb FAIL (2025-12-29 10:41:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 10:41:30.347434087 (571h12m30.218517122s) [upload=500781]: Starting upload Dec 29 10:41:30.347488100 (571h12m30.218571135s) [upload=500781]: Done starting upload Dec 29 10:41:30.347621504 (571h12m30.218704539s) [upload=500781]: Uploading segments... Dec 29 10:41:30.347853826 (571h12m30.218936861s) [upload=500781]: Got next segment. Inline: false Dec 29 10:41:30.347940231 (571h12m30.219023296s) [upload=500781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:41:30.347948407 (571h12m30.219031482s) [upload=500781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:41:30.591818362 (571h12m30.462901407s) [seg_pos={0 0}, upload=500781]: Begin upload segment... Dec 29 10:41:30.591874830 (571h12m30.462957865s) [seg_pos={0 0}, upload=500781]: Done begin upload segment. Dec 29 10:41:30.591903534 (571h12m30.462986569s) [noise=true , address=45.91.116.14:4191 , piece_id=53I2LCL3NV3W2D3AOJRNNPURRILPCVBFIVZBEMCJ7CYHXMAXBDHQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.591959941 (571h12m30.463042986s) [noise=true , address=45.91.116.11:4117 , piece_id=P5JTGCXCI4W2ZJBCUL4QHVVJFGYKNO3RVONGHBK5HMYIJABGJHMQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.592996315 (571h12m30.46407935s) [noise=true , address=45.91.116.7:4010 , piece_id=3S3LUIJAJPBTAYWFEU2TNFKG2WY2UD65SR4DN5V7PZL4G2BGAN7Q, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.593374766 (571h12m30.464457811s) [noise=true , address=145.79.227.7:4295 , piece_id=ICOYDH46Z56S4ZOEPPLBJKXLSXKNDLWOMV45R7C2MD3VJQAYDWAA, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.593494162 (571h12m30.464577208s) [noise=true , address=145.79.227.3:4150 , piece_id=BBKVIVMTZDAXTUR5RNFVOWARE6EFVUB2SO2TUV2IC6Q3Y4EG6PGA, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.593918600 (571h12m30.465001636s) [noise=true , address=145.79.227.6:4259 , piece_id=YMIWZ4JM7IQIDWJYZSJJK5PHX5YSYE5N3DPAOV2ZNBTYS67EBHYA, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.594069709 (571h12m30.465152744s) [noise=true , address=145.79.227.19:4078 , piece_id=FQZMQ3SRRRP5J5V3UKJLDFGCUZP4RLBT6MKV434PJEIH7WA6SYQQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.594260993 (571h12m30.465344028s) [noise=true , address=145.79.227.5:4222 , piece_id=F5QIXFKE6CPEBMFKG2AOSOJXXAD7UC7LUQGAXJ3OFDSWBBGMKCNQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.594461354 (571h12m30.465544389s) [noise=true , address=145.79.227.45:4229 , piece_id=YUIB2ROL44POFWTD2RG6L75NAY5VGM4QZUOOKH33CLMPVOSVTY3A, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.594706661 (571h12m30.465789697s) [noise=true , address=145.79.227.39:4015 , piece_id=NEHIVNH3QCS33J4CSDJBZI6UF43NKKCEE6CF6IMOQTD23Q2CNGWA, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.594871866 (571h12m30.465954901s) [noise=true , address=145.79.227.49:4337 , piece_id=X324WWTFFWS7XZPVDLHUIKTDHXWN4QCBOKYMCWW5GTI2C4WBGCEQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.595094871 (571h12m30.466177906s) [noise=true , address=45.91.116.4:4939 , piece_id=H6AHECGUMDBLJE7VXM56JUNRDWTI46TAYCZIEPSUAPE6T45Q4JMA, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.595164112 (571h12m30.466247147s) [noise=true , address=145.79.227.41:4085 , piece_id=3WYKT6X37COWTGJGNFFJPWYM56LL7UTG3YRZIEZ34KGI24YYM7AQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.595511514 (571h12m30.466594559s) [noise=true , address=145.79.227.13:4804 , piece_id=H2LLMN74VH56GXWNL3ZXQX57CRKW6YMO2G3IRQHSS6A2QSORLNUA, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.596094324 (571h12m30.467177369s) [noise=true , address=77.90.31.13:4629 , piece_id=GRYRYG2LP3AMMJZM2PEIMAW5VHFHUNBYB6D33YIRYT72RYUF5YHQ, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:30.596588505 (571h12m30.46767154s) [noise=true , address=45.91.116.13:4154 , piece_id=Z6X3CA2SI4ZXEOD3NNKAYEPYM4YT3SKAYM4Q5QIXZDW2JKICVH7A, seg_pos={0 0}, upload=500781]: Uploading piece... Dec 29 10:41:32.200459029 (571h12m32.071542074s) [upload=500781]: Next returned nil segment Dec 29 10:41:32.200475501 (571h12m32.071558546s) [upload=500781]: Waiting for error group managing segments... Dec 29 10:41:32.580861003 (571h12m32.451944049s) [noise=true , address=145.79.227.6:4259 , piece_id=YMIWZ4JM7IQIDWJYZSJJK5PHX5YSYE5N3DPAOV2ZNBTYS67EBHYA, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.585116066 (571h12m32.456199111s) [noise=true , address=145.79.227.49:4337 , piece_id=X324WWTFFWS7XZPVDLHUIKTDHXWN4QCBOKYMCWW5GTI2C4WBGCEQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.586109176 (571h12m32.457192211s) [noise=true , address=145.79.227.19:4078 , piece_id=FQZMQ3SRRRP5J5V3UKJLDFGCUZP4RLBT6MKV434PJEIH7WA6SYQQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.586614379 (571h12m32.457697414s) [noise=true , address=145.79.227.13:4804 , piece_id=H2LLMN74VH56GXWNL3ZXQX57CRKW6YMO2G3IRQHSS6A2QSORLNUA, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.586692838 (571h12m32.457775883s) [noise=true , address=145.79.227.5:4222 , piece_id=F5QIXFKE6CPEBMFKG2AOSOJXXAD7UC7LUQGAXJ3OFDSWBBGMKCNQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.588456136 (571h12m32.459539191s) [noise=true , address=145.79.227.3:4150 , piece_id=BBKVIVMTZDAXTUR5RNFVOWARE6EFVUB2SO2TUV2IC6Q3Y4EG6PGA, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.589401977 (571h12m32.460485022s) [noise=true , address=145.79.227.41:4085 , piece_id=3WYKT6X37COWTGJGNFFJPWYM56LL7UTG3YRZIEZ34KGI24YYM7AQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.595312612 (571h12m32.466395667s) [noise=true , address=145.79.227.45:4229 , piece_id=YUIB2ROL44POFWTD2RG6L75NAY5VGM4QZUOOKH33CLMPVOSVTY3A, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.606470307 (571h12m32.477553352s) [noise=true , address=145.79.227.39:4015 , piece_id=NEHIVNH3QCS33J4CSDJBZI6UF43NKKCEE6CF6IMOQTD23Q2CNGWA, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.618836004 (571h12m32.489919039s) [noise=true , address=145.79.227.7:4295 , piece_id=ICOYDH46Z56S4ZOEPPLBJKXLSXKNDLWOMV45R7C2MD3VJQAYDWAA, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.790301209 (571h12m32.661384244s) [noise=true , address=77.90.31.13:4629 , piece_id=GRYRYG2LP3AMMJZM2PEIMAW5VHFHUNBYB6D33YIRYT72RYUF5YHQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.827468461 (571h12m32.698551507s) [noise=true , address=45.91.116.7:4010 , piece_id=3S3LUIJAJPBTAYWFEU2TNFKG2WY2UD65SR4DN5V7PZL4G2BGAN7Q, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.832258403 (571h12m32.703341448s) [noise=true , address=45.91.116.14:4191 , piece_id=53I2LCL3NV3W2D3AOJRNNPURRILPCVBFIVZBEMCJ7CYHXMAXBDHQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: <nil> Dec 29 10:41:32.832307516 (571h12m32.703390551s) [seg_pos={0 0}, upload=500781]: Segment reached optimal threshold of 13 pieces. Dec 29 10:41:32.832658985 (571h12m32.70374203s) [noise=true , address=45.91.116.11:4117 , piece_id=P5JTGCXCI4W2ZJBCUL4QHVVJFGYKNO3RVONGHBK5HMYIJABGJHMQ, seg_pos={0 0}, upload=500781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:37930->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:37930->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37930->45.91.116.11:4117: use of closed network connection Dec 29 10:41:32.832674304 (571h12m32.703757369s) [noise=true , address=45.91.116.11:4117 , piece_id=P5JTGCXCI4W2ZJBCUL4QHVVJFGYKNO3RVONGHBK5HMYIJABGJHMQ, seg_pos={0 0}, upload=500781]: Long tail cancellation due to optimal threshold Dec 29 10:41:32.833185758 (571h12m32.704268803s) [noise=true , address=45.91.116.4:4939 , piece_id=H6AHECGUMDBLJE7VXM56JUNRDWTI46TAYCZIEPSUAPE6T45Q4JMA, seg_pos={0 0}, upload=500781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 10:41:32.833194314 (571h12m32.704277359s) [noise=true , address=45.91.116.4:4939 , piece_id=H6AHECGUMDBLJE7VXM56JUNRDWTI46TAYCZIEPSUAPE6T45Q4JMA, seg_pos={0 0}, upload=500781]: Long tail cancellation due to optimal threshold Dec 29 10:41:32.833401699 (571h12m32.704484734s) [noise=true , address=45.91.116.13:4154 , piece_id=Z6X3CA2SI4ZXEOD3NNKAYEPYM4YT3SKAYM4Q5QIXZDW2JKICVH7A, seg_pos={0 0}, upload=500781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 10:41:32.833408212 (571h12m32.704491247s) [noise=true , address=45.91.116.13:4154 , piece_id=Z6X3CA2SI4ZXEOD3NNKAYEPYM4YT3SKAYM4Q5QIXZDW2JKICVH7A, seg_pos={0 0}, upload=500781]: Long tail cancellation due to optimal threshold Dec 29 10:41:32.833450432 (571h12m32.704533477s) [seg_pos={0 0}, upload=500781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:41:32.833574718 (571h12m32.704657753s) [upload=500781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:41:32.833586151 (571h12m32.704669186s) [upload=500781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:41:33.129930155 (571h12m33.0010132s) [upload=500781]: Done uploading segments... Dec 29 10:41:33.129954811 (571h12m33.001037856s) [upload=500781]: Upload finished. err: <nil> Duration: 30.004014033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.......................................X......] duration: 54.595275788s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:41:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 54.595292539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......................................X.....] duration: 48.463723968s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:41:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 48.46374666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X......X..X.......X....X..............X...X.X.X....X...XXX..] duration: 3.46257488s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:41:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462610708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X...X..XX...X..XX..X......X.X.X..X..........X...X..X....] duration: 5.707499752s trace: https://jaeger.dev.storj.io/trace/6568dba7c8538f89 FAIL (2025-12-29 10:40:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ) context deadline exceeded ======LOGS====== Dec 29 10:38:15.102950520 (571h9m14.974033575s) [upload=500732]: Starting upload Dec 29 10:38:15.103074956 (571h9m14.974157991s) [upload=500732]: Done starting upload Dec 29 10:38:15.103169537 (571h9m14.974252582s) [upload=500732]: Uploading segments... Dec 29 10:38:15.103465500 (571h9m14.974548535s) [upload=500732]: Got next segment. Inline: false Dec 29 10:38:15.103515015 (571h9m14.97459806s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:38:15.103520665 (571h9m14.97460372s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:38:15.360755610 (571h9m15.231838655s) [seg_pos={0 0}, upload=500732]: Begin upload segment... Dec 29 10:38:15.360793262 (571h9m15.231876307s) [seg_pos={0 0}, upload=500732]: Done begin upload segment. Dec 29 10:38:15.360816055 (571h9m15.2318991s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.363112648 (571h9m15.234195693s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.364664513 (571h9m15.235747548s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.365710365 (571h9m15.23679341s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.366067014 (571h9m15.237150049s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367643555 (571h9m15.2387266s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367833357 (571h9m15.238916402s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369695984 (571h9m15.240779019s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369746389 (571h9m15.240829424s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371646678 (571h9m15.242729713s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371655835 (571h9m15.24273888s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.373722090 (571h9m15.244805125s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.374522174 (571h9m15.245605209s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.375360019 (571h9m15.246443054s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.376683750 (571h9m15.247766795s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.379095191 (571h9m15.250178236s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:17.069983781 (571h9m16.941066816s) [upload=500732]: Next returned nil segment Dec 29 10:38:17.070017085 (571h9m16.94110012s) [upload=500732]: Waiting for error group managing segments... Dec 29 10:38:17.359388843 (571h9m17.230471888s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.359846695 (571h9m17.23092973s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.367882136 (571h9m17.238965171s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.368160446 (571h9m17.239243481s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.389071082 (571h9m17.260154127s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.573261259 (571h9m17.444344294s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.605706339 (571h9m17.476789384s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.617366121 (571h9m17.488449156s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.680445915 (571h9m17.55152897s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.881112414 (571h9m17.752195449s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.959026725 (571h9m17.83010977s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.318939404 (571h9m18.190022439s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326374742 (571h9m18.197457787s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326402024 (571h9m18.197485069s) [seg_pos={0 0}, upload=500732]: Segment reached optimal threshold of 13 pieces. Dec 29 10:38:18.326628586 (571h9m18.197711631s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection Dec 29 10:38:18.326637302 (571h9m18.197720347s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326688309 (571h9m18.197771344s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326704220 (571h9m18.197787255s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326821382 (571h9m18.197904417s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326826923 (571h9m18.197909968s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326848013 (571h9m18.197931048s) [seg_pos={0 0}, upload=500732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:38:18.326957281 (571h9m18.198040316s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:38:18.326963623 (571h9m18.198046658s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:38:18.617479670 (571h9m18.488562705s) [upload=500732]: Done uploading segments... Dec 29 10:38:18.617500469 (571h9m18.488583524s) [upload=500732]: Upload finished. err: <nil> Duration: 5.707528918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......................................X....] duration: 36.058821831s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:40:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 36.058855305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X...X..XX...X..XX..X......X.X.X..X..........X...X..X...] duration: 6.873173557s trace: https://jaeger.dev.storj.io/trace/6568dba7c8538f89 FAIL (2025-12-29 10:40:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ) context deadline exceeded ======LOGS====== Dec 29 10:38:15.102950520 (571h9m14.974033575s) [upload=500732]: Starting upload Dec 29 10:38:15.103074956 (571h9m14.974157991s) [upload=500732]: Done starting upload Dec 29 10:38:15.103169537 (571h9m14.974252582s) [upload=500732]: Uploading segments... Dec 29 10:38:15.103465500 (571h9m14.974548535s) [upload=500732]: Got next segment. Inline: false Dec 29 10:38:15.103515015 (571h9m14.97459806s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:38:15.103520665 (571h9m14.97460372s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:38:15.360755610 (571h9m15.231838655s) [seg_pos={0 0}, upload=500732]: Begin upload segment... Dec 29 10:38:15.360793262 (571h9m15.231876307s) [seg_pos={0 0}, upload=500732]: Done begin upload segment. Dec 29 10:38:15.360816055 (571h9m15.2318991s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.363112648 (571h9m15.234195693s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.364664513 (571h9m15.235747548s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.365710365 (571h9m15.23679341s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.366067014 (571h9m15.237150049s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367643555 (571h9m15.2387266s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367833357 (571h9m15.238916402s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369695984 (571h9m15.240779019s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369746389 (571h9m15.240829424s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371646678 (571h9m15.242729713s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371655835 (571h9m15.24273888s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.373722090 (571h9m15.244805125s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.374522174 (571h9m15.245605209s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.375360019 (571h9m15.246443054s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.376683750 (571h9m15.247766795s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.379095191 (571h9m15.250178236s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:17.069983781 (571h9m16.941066816s) [upload=500732]: Next returned nil segment Dec 29 10:38:17.070017085 (571h9m16.94110012s) [upload=500732]: Waiting for error group managing segments... Dec 29 10:38:17.359388843 (571h9m17.230471888s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.359846695 (571h9m17.23092973s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.367882136 (571h9m17.238965171s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.368160446 (571h9m17.239243481s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.389071082 (571h9m17.260154127s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.573261259 (571h9m17.444344294s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.605706339 (571h9m17.476789384s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.617366121 (571h9m17.488449156s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.680445915 (571h9m17.55152897s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.881112414 (571h9m17.752195449s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.959026725 (571h9m17.83010977s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.318939404 (571h9m18.190022439s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326374742 (571h9m18.197457787s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326402024 (571h9m18.197485069s) [seg_pos={0 0}, upload=500732]: Segment reached optimal threshold of 13 pieces. Dec 29 10:38:18.326628586 (571h9m18.197711631s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection Dec 29 10:38:18.326637302 (571h9m18.197720347s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326688309 (571h9m18.197771344s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326704220 (571h9m18.197787255s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326821382 (571h9m18.197904417s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326826923 (571h9m18.197909968s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326848013 (571h9m18.197931048s) [seg_pos={0 0}, upload=500732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:38:18.326957281 (571h9m18.198040316s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:38:18.326963623 (571h9m18.198046658s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:38:18.617479670 (571h9m18.488562705s) [upload=500732]: Done uploading segments... Dec 29 10:38:18.617500469 (571h9m18.488583524s) [upload=500732]: Upload finished. err: <nil> Duration: 6.873204336s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX......X..X.......X....X..............X...X.X.X....X...XXX.] duration: 4.757646776s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:39:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.757783426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXX......X..X.......X....X..............X...X.X.X....X...XXX] duration: 5.001227713s trace: https://jaeger.dev.storj.io/trace/4631881f0c2a64fa FAIL (2025-12-29 10:39:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001247059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....X...X..XX...X..XX..X......X.X.X..X..........X...X..X..] duration: 6.012559215s trace: https://jaeger.dev.storj.io/trace/6568dba7c8538f89 FAIL (2025-12-29 10:39:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ) context deadline exceeded ======LOGS====== Dec 29 10:38:15.102950520 (571h9m14.974033575s) [upload=500732]: Starting upload Dec 29 10:38:15.103074956 (571h9m14.974157991s) [upload=500732]: Done starting upload Dec 29 10:38:15.103169537 (571h9m14.974252582s) [upload=500732]: Uploading segments... Dec 29 10:38:15.103465500 (571h9m14.974548535s) [upload=500732]: Got next segment. Inline: false Dec 29 10:38:15.103515015 (571h9m14.97459806s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:38:15.103520665 (571h9m14.97460372s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:38:15.360755610 (571h9m15.231838655s) [seg_pos={0 0}, upload=500732]: Begin upload segment... Dec 29 10:38:15.360793262 (571h9m15.231876307s) [seg_pos={0 0}, upload=500732]: Done begin upload segment. Dec 29 10:38:15.360816055 (571h9m15.2318991s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.363112648 (571h9m15.234195693s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.364664513 (571h9m15.235747548s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.365710365 (571h9m15.23679341s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.366067014 (571h9m15.237150049s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367643555 (571h9m15.2387266s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367833357 (571h9m15.238916402s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369695984 (571h9m15.240779019s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369746389 (571h9m15.240829424s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371646678 (571h9m15.242729713s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371655835 (571h9m15.24273888s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.373722090 (571h9m15.244805125s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.374522174 (571h9m15.245605209s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.375360019 (571h9m15.246443054s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.376683750 (571h9m15.247766795s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.379095191 (571h9m15.250178236s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:17.069983781 (571h9m16.941066816s) [upload=500732]: Next returned nil segment Dec 29 10:38:17.070017085 (571h9m16.94110012s) [upload=500732]: Waiting for error group managing segments... Dec 29 10:38:17.359388843 (571h9m17.230471888s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.359846695 (571h9m17.23092973s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.367882136 (571h9m17.238965171s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.368160446 (571h9m17.239243481s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.389071082 (571h9m17.260154127s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.573261259 (571h9m17.444344294s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.605706339 (571h9m17.476789384s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.617366121 (571h9m17.488449156s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.680445915 (571h9m17.55152897s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.881112414 (571h9m17.752195449s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.959026725 (571h9m17.83010977s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.318939404 (571h9m18.190022439s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326374742 (571h9m18.197457787s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326402024 (571h9m18.197485069s) [seg_pos={0 0}, upload=500732]: Segment reached optimal threshold of 13 pieces. Dec 29 10:38:18.326628586 (571h9m18.197711631s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection Dec 29 10:38:18.326637302 (571h9m18.197720347s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326688309 (571h9m18.197771344s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326704220 (571h9m18.197787255s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326821382 (571h9m18.197904417s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326826923 (571h9m18.197909968s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326848013 (571h9m18.197931048s) [seg_pos={0 0}, upload=500732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:38:18.326957281 (571h9m18.198040316s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:38:18.326963623 (571h9m18.198046658s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:38:18.617479670 (571h9m18.488562705s) [upload=500732]: Done uploading segments... Dec 29 10:38:18.617500469 (571h9m18.488583524s) [upload=500732]: Upload finished. err: <nil> Duration: 6.012590716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X....X...X..XX...X..XX..X......X.X.X..X..........X...X..X.] duration: 5.937238339s trace: https://jaeger.dev.storj.io/trace/6568dba7c8538f89 FAIL (2025-12-29 10:39:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ) context deadline exceeded ======LOGS====== Dec 29 10:38:15.102950520 (571h9m14.974033575s) [upload=500732]: Starting upload Dec 29 10:38:15.103074956 (571h9m14.974157991s) [upload=500732]: Done starting upload Dec 29 10:38:15.103169537 (571h9m14.974252582s) [upload=500732]: Uploading segments... Dec 29 10:38:15.103465500 (571h9m14.974548535s) [upload=500732]: Got next segment. Inline: false Dec 29 10:38:15.103515015 (571h9m14.97459806s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:38:15.103520665 (571h9m14.97460372s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:38:15.360755610 (571h9m15.231838655s) [seg_pos={0 0}, upload=500732]: Begin upload segment... Dec 29 10:38:15.360793262 (571h9m15.231876307s) [seg_pos={0 0}, upload=500732]: Done begin upload segment. Dec 29 10:38:15.360816055 (571h9m15.2318991s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.363112648 (571h9m15.234195693s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.364664513 (571h9m15.235747548s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.365710365 (571h9m15.23679341s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.366067014 (571h9m15.237150049s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367643555 (571h9m15.2387266s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367833357 (571h9m15.238916402s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369695984 (571h9m15.240779019s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369746389 (571h9m15.240829424s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371646678 (571h9m15.242729713s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371655835 (571h9m15.24273888s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.373722090 (571h9m15.244805125s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.374522174 (571h9m15.245605209s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.375360019 (571h9m15.246443054s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.376683750 (571h9m15.247766795s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.379095191 (571h9m15.250178236s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:17.069983781 (571h9m16.941066816s) [upload=500732]: Next returned nil segment Dec 29 10:38:17.070017085 (571h9m16.94110012s) [upload=500732]: Waiting for error group managing segments... Dec 29 10:38:17.359388843 (571h9m17.230471888s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.359846695 (571h9m17.23092973s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.367882136 (571h9m17.238965171s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.368160446 (571h9m17.239243481s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.389071082 (571h9m17.260154127s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.573261259 (571h9m17.444344294s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.605706339 (571h9m17.476789384s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.617366121 (571h9m17.488449156s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.680445915 (571h9m17.55152897s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.881112414 (571h9m17.752195449s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.959026725 (571h9m17.83010977s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.318939404 (571h9m18.190022439s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326374742 (571h9m18.197457787s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326402024 (571h9m18.197485069s) [seg_pos={0 0}, upload=500732]: Segment reached optimal threshold of 13 pieces. Dec 29 10:38:18.326628586 (571h9m18.197711631s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection Dec 29 10:38:18.326637302 (571h9m18.197720347s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326688309 (571h9m18.197771344s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326704220 (571h9m18.197787255s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326821382 (571h9m18.197904417s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326826923 (571h9m18.197909968s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326848013 (571h9m18.197931048s) [seg_pos={0 0}, upload=500732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:38:18.326957281 (571h9m18.198040316s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:38:18.326963623 (571h9m18.198046658s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:38:18.617479670 (571h9m18.488562705s) [upload=500732]: Done uploading segments... Dec 29 10:38:18.617500469 (571h9m18.488583524s) [upload=500732]: Upload finished. err: <nil> Duration: 5.937295489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XXX......X..X.......X....X..............X...X.X.X....X...XX] duration: 5.0006647s trace: https://jaeger.dev.storj.io/trace/174e73f45c4f8203 FAIL (2025-12-29 10:39:17 UTC) orbiter warning: context deadline exceeded Duration: 5.000684538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.......................................X...] duration: 24.274403154s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 24.274481723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X....X...X..XX...X..XX..X......X.X.X..X..........X...X..X] duration: 30.002231375s trace: https://jaeger.dev.storj.io/trace/6568dba7c8538f89 FAIL (2025-12-29 10:38:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ) context deadline exceeded ======LOGS====== Dec 29 10:38:15.102950520 (571h9m14.974033575s) [upload=500732]: Starting upload Dec 29 10:38:15.103074956 (571h9m14.974157991s) [upload=500732]: Done starting upload Dec 29 10:38:15.103169537 (571h9m14.974252582s) [upload=500732]: Uploading segments... Dec 29 10:38:15.103465500 (571h9m14.974548535s) [upload=500732]: Got next segment. Inline: false Dec 29 10:38:15.103515015 (571h9m14.97459806s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:38:15.103520665 (571h9m14.97460372s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:38:15.360755610 (571h9m15.231838655s) [seg_pos={0 0}, upload=500732]: Begin upload segment... Dec 29 10:38:15.360793262 (571h9m15.231876307s) [seg_pos={0 0}, upload=500732]: Done begin upload segment. Dec 29 10:38:15.360816055 (571h9m15.2318991s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.363112648 (571h9m15.234195693s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.364664513 (571h9m15.235747548s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.365710365 (571h9m15.23679341s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.366067014 (571h9m15.237150049s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367643555 (571h9m15.2387266s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.367833357 (571h9m15.238916402s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369695984 (571h9m15.240779019s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.369746389 (571h9m15.240829424s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371646678 (571h9m15.242729713s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.371655835 (571h9m15.24273888s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.373722090 (571h9m15.244805125s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.374522174 (571h9m15.245605209s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.375360019 (571h9m15.246443054s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.376683750 (571h9m15.247766795s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:15.379095191 (571h9m15.250178236s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Uploading piece... Dec 29 10:38:17.069983781 (571h9m16.941066816s) [upload=500732]: Next returned nil segment Dec 29 10:38:17.070017085 (571h9m16.94110012s) [upload=500732]: Waiting for error group managing segments... Dec 29 10:38:17.359388843 (571h9m17.230471888s) [noise=true , address=145.79.227.24:4582 , piece_id=DRIFEIWP6JWVWYRUYE2HU6DKN2YIKLDEM24K6DSGNZOHAXIFLDLA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.359846695 (571h9m17.23092973s) [noise=true , address=145.79.227.5:4222 , piece_id=WD2CLAMBVYDYCVVNATE2ZZAU4BHZWTL4VGEWDJFOXJLUQXCR2KRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.367882136 (571h9m17.238965171s) [noise=true , address=145.79.227.41:4086 , piece_id=HUXNE76CPRINFDTN2EUZ5FLFWHHNGDPO3J6XNEWCF5EJWWZNKIBA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.368160446 (571h9m17.239243481s) [noise=true , address=145.79.227.36:4517 , piece_id=VLVMRKW7REPERNEVTTTMII4BNUV36FKIFSJX7XJPC4MQXWOGVCVA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.389071082 (571h9m17.260154127s) [noise=true , address=145.79.227.16:4911 , piece_id=5IU66AKP5G76B7TB5GCI4673ECQJ5AM7QTD6JMEV62WJSCMTSNTQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.573261259 (571h9m17.444344294s) [noise=true , address=77.90.31.15:4697 , piece_id=GZPV25NQEO5H4EKBOMVWRAHZYFZMO346GZA3DV46XIBOFHSBG3XA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.605706339 (571h9m17.476789384s) [noise=true , address=45.91.116.11:4118 , piece_id=PPRT73IDRERAEUQQUCPQPKTWSOAOHH3BSJ4MLPPETV6FT2OOFYXA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.617366121 (571h9m17.488449156s) [noise=true , address=45.91.116.8:4047 , piece_id=3JBU7PADHQMCD3WHSJ4MZRMON7K2S2FTJBE56WQM46LV7PGHV6HQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.680445915 (571h9m17.55152897s) [noise=true , address=77.90.31.18:4808 , piece_id=IAG7M2GFTU7W4ILPQXF34PS35JVEWMX2DEPJVEXIS4DN2MMTLKRQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.881112414 (571h9m17.752195449s) [noise=true , address=77.90.31.14:4662 , piece_id=ABWVRK2C6HNKZ5BSESIQGGGLKAJGUELGMYFBSLIQCM6YU4Y6KFUQ, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:17.959026725 (571h9m17.83010977s) [noise=true , address=45.91.116.6:4974 , piece_id=SSSI6JNR4MVTGCNVUC3ZPTG2KKAYFR6KVFKEYABH5SIKCLUD7JJA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.318939404 (571h9m18.190022439s) [noise=true , address=45.91.116.7:4010 , piece_id=GRLJWPHNGFMDIBKSHW57RHBL33CT7AUKNL44K4IZ77TD6FSNIG4Q, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326374742 (571h9m18.197457787s) [noise=true , address=45.91.116.14:4192 , piece_id=4TT3O2ZRCY77JTDFNYMLBGKZ7P5DH6TIBMHWXEYQU3VV5PMYTJOA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: <nil> Dec 29 10:38:18.326402024 (571h9m18.197485069s) [seg_pos={0 0}, upload=500732]: Segment reached optimal threshold of 13 pieces. Dec 29 10:38:18.326628586 (571h9m18.197711631s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60500->77.90.31.10:4517: use of closed network connection Dec 29 10:38:18.326637302 (571h9m18.197720347s) [noise=true , address=77.90.31.10:4517 , piece_id=DUUAXA32M7GBTPY3BTP57BRMCVIT2ILNACG4HE66YER5CK5P7BNA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326688309 (571h9m18.197771344s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326704220 (571h9m18.197787255s) [noise=true , address=145.79.227.44:4193 , piece_id=FDVI5H72X2KGDWPUIXCFJSOSWPAEEZF7YRXXJ7L2FSAFP5V254VA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326821382 (571h9m18.197904417s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 10:38:18.326826923 (571h9m18.197909968s) [noise=true , address=77.90.31.11:4556 , piece_id=FUQI2TATN7D2VORPAZOZZZY37TTICLBYRRWAOHY3RJON2QWKEDQA, seg_pos={0 0}, upload=500732]: Long tail cancellation due to optimal threshold Dec 29 10:38:18.326848013 (571h9m18.197931048s) [seg_pos={0 0}, upload=500732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:38:18.326957281 (571h9m18.198040316s) [upload=500732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:38:18.326963623 (571h9m18.198046658s) [upload=500732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:38:18.617479670 (571h9m18.488562705s) [upload=500732]: Done uploading segments... Dec 29 10:38:18.617500469 (571h9m18.488583524s) [upload=500732]: Upload finished. err: <nil> Duration: 30.002252876s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XXX......X..X.......X....X..............X...X.X.X....X...X] duration: 5.001559657s trace: https://jaeger.dev.storj.io/trace/35fc206d785434a9 FAIL (2025-12-29 10:38:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001583994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......................................X..] duration: 20.41301799s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:38:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 20.413033168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......................................X.] duration: 36.465310981s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:37:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 36.465340257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X....X...X..XX...X..XX..X......X.X.X..X..........X...X..] duration: 5.943641868s trace: https://jaeger.dev.storj.io/trace/bd2bae9b250254b FAIL (2025-12-29 10:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA) context deadline exceeded ======LOGS====== Dec 29 10:36:02.516563031 (571h7m2.387646076s) [upload=500698]: Starting upload Dec 29 10:36:02.516647832 (571h7m2.387730877s) [upload=500698]: Done starting upload Dec 29 10:36:02.516725099 (571h7m2.387808144s) [upload=500698]: Uploading segments... Dec 29 10:36:02.517074164 (571h7m2.388157219s) [upload=500698]: Got next segment. Inline: false Dec 29 10:36:02.517125212 (571h7m2.388208267s) [upload=500698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:36:02.517130832 (571h7m2.388213877s) [upload=500698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:36:02.765279145 (571h7m2.63636219s) [seg_pos={0 0}, upload=500698]: Begin upload segment... Dec 29 10:36:02.765338348 (571h7m2.636421393s) [seg_pos={0 0}, upload=500698]: Done begin upload segment. Dec 29 10:36:02.765374908 (571h7m2.636457953s) [noise=true , address=145.79.227.14:4839 , piece_id=N4U5MSRE3MSWRIOYKGLPHDXJMOWHXU6XC2MQTI53PZCN23TP2LTA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.765523049 (571h7m2.636606084s) [noise=true , address=145.79.227.38:4977 , piece_id=3QAS32F2RSMIQFAEAVANSP5MCVNAYGJTNDW7ZCGDDIORCOGLTBKA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.766436089 (571h7m2.637519134s) [noise=true , address=77.90.31.12:4589 , piece_id=MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.766943835 (571h7m2.63802689s) [noise=true , address=77.90.31.16:4735 , piece_id=J4NKX4S3WEQZ75P2QAGNHIV76PG7BKHEW7K5IJUPHVIIOCN6WVYA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.767456161 (571h7m2.638539206s) [noise=true , address=45.91.116.8:4047 , piece_id=OFQDGRAZOUBHEZ7JOQCVAZMIZPCGPKZPCNAB7WXDAI44NZASWO7Q, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.767917749 (571h7m2.639000795s) [noise=true , address=45.91.116.7:4010 , piece_id=7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.768387594 (571h7m2.639470639s) [noise=true , address=145.79.227.20:4437 , piece_id=YCOJURIYPT7A736I7RBYZOWYDISPVDPUZQ2P5Q5FEHWPUCFQUHLA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.768874020 (571h7m2.639957065s) [noise=true , address=145.79.227.7:4293 , piece_id=FGTHHPPEY4SJYAJYGVV6P6U6LOXVEIYGE7COZ2M7IUI73QJTRZMQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.770931558 (571h7m2.642014613s) [noise=true , address=145.79.227.39:4013 , piece_id=VMK54RDXXLRO5JFPHXY65XOTQGFPISLMGNU4GAHFUUCIVRHTY5IA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.771934588 (571h7m2.643017633s) [noise=true , address=145.79.227.6:4259 , piece_id=QSQWMG2EQMIRFSDK2B6ODOLYBTYPCNSZAXK6DP2L7F5VPHBBN3IQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.773055302 (571h7m2.644138347s) [noise=true , address=77.90.31.13:4629 , piece_id=H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.775107159 (571h7m2.646190214s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.776727263 (571h7m2.647810308s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.778602304 (571h7m2.649685339s) [noise=true , address=145.79.227.45:4229 , piece_id=4PWMCZW7I7IONT56GD43XWH4OOYXGLCFEPWKEXA5DPBSIQGLMMPQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.780232248 (571h7m2.651315293s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.788867992 (571h7m2.659951037s) [noise=true , address=145.79.227.19:4078 , piece_id=5QG2HNCMSQ7NGWLZ736KWLOVP5UP3NSVWOUYBV6J2COYOQ4U4FTQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:04.377125647 (571h7m4.248208682s) [upload=500698]: Next returned nil segment Dec 29 10:36:04.377141577 (571h7m4.248224642s) [upload=500698]: Waiting for error group managing segments... Dec 29 10:36:04.757104383 (571h7m4.628187428s) [noise=true , address=145.79.227.20:4437 , piece_id=YCOJURIYPT7A736I7RBYZOWYDISPVDPUZQ2P5Q5FEHWPUCFQUHLA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.762309806 (571h7m4.633392851s) [noise=true , address=145.79.227.45:4229 , piece_id=4PWMCZW7I7IONT56GD43XWH4OOYXGLCFEPWKEXA5DPBSIQGLMMPQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.764708964 (571h7m4.635792009s) [noise=true , address=145.79.227.38:4977 , piece_id=3QAS32F2RSMIQFAEAVANSP5MCVNAYGJTNDW7ZCGDDIORCOGLTBKA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.765123674 (571h7m4.636206709s) [noise=true , address=145.79.227.6:4259 , piece_id=QSQWMG2EQMIRFSDK2B6ODOLYBTYPCNSZAXK6DP2L7F5VPHBBN3IQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.771422879 (571h7m4.642505924s) [noise=true , address=145.79.227.14:4839 , piece_id=N4U5MSRE3MSWRIOYKGLPHDXJMOWHXU6XC2MQTI53PZCN23TP2LTA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.773345269 (571h7m4.644428314s) [noise=true , address=145.79.227.19:4078 , piece_id=5QG2HNCMSQ7NGWLZ736KWLOVP5UP3NSVWOUYBV6J2COYOQ4U4FTQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.773381989 (571h7m4.644465034s) [noise=true , address=145.79.227.39:4013 , piece_id=VMK54RDXXLRO5JFPHXY65XOTQGFPISLMGNU4GAHFUUCIVRHTY5IA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.787804893 (571h7m4.658887938s) [noise=true , address=145.79.227.7:4293 , piece_id=FGTHHPPEY4SJYAJYGVV6P6U6LOXVEIYGE7COZ2M7IUI73QJTRZMQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.022385700 (571h7m4.893468745s) [noise=true , address=45.91.116.7:4010 , piece_id=7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.036273144 (571h7m4.907356189s) [noise=true , address=77.90.31.13:4629 , piece_id=H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.038862995 (571h7m4.909946041s) [noise=true , address=45.91.116.8:4047 , piece_id=OFQDGRAZOUBHEZ7JOQCVAZMIZPCGPKZPCNAB7WXDAI44NZASWO7Q, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.108104462 (571h7m4.979187507s) [noise=true , address=77.90.31.16:4735 , piece_id=J4NKX4S3WEQZ75P2QAGNHIV76PG7BKHEW7K5IJUPHVIIOCN6WVYA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.155902817 (571h7m5.026985862s) [noise=true , address=77.90.31.12:4589 , piece_id=MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.155931702 (571h7m5.027014748s) [seg_pos={0 0}, upload=500698]: Segment reached optimal threshold of 13 pieces. Dec 29 10:36:05.156099362 (571h7m5.027182407s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:05.156106034 (571h7m5.027189079s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156382421 (571h7m5.027465496s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection Dec 29 10:36:05.156395575 (571h7m5.027478651s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156547274 (571h7m5.027630309s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:05.156553787 (571h7m5.027636822s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156571721 (571h7m5.027654756s) [seg_pos={0 0}, upload=500698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:36:05.156634861 (571h7m5.027717906s) [upload=500698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:05.156639560 (571h7m5.027722605s) [upload=500698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:05.444302965 (571h7m5.315386s) [upload=500698]: Done uploading segments... Dec 29 10:36:05.444323063 (571h7m5.315406098s) [upload=500698]: Upload finished. err: <nil> Duration: 5.943669992s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X..XXX......X..X.......X....X..............X...X.X.X....X...] duration: 5.000698506s trace: https://jaeger.dev.storj.io/trace/60335edd44b6c64f FAIL (2025-12-29 10:37:14 UTC) orbiter warning: context deadline exceeded Duration: 5.000721409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X..X....X...X..XX...X..XX..X......X.X.X..X..........X...X.] duration: 5.740181142s trace: https://jaeger.dev.storj.io/trace/bd2bae9b250254b FAIL (2025-12-29 10:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA) context deadline exceeded ======LOGS====== Dec 29 10:36:02.516563031 (571h7m2.387646076s) [upload=500698]: Starting upload Dec 29 10:36:02.516647832 (571h7m2.387730877s) [upload=500698]: Done starting upload Dec 29 10:36:02.516725099 (571h7m2.387808144s) [upload=500698]: Uploading segments... Dec 29 10:36:02.517074164 (571h7m2.388157219s) [upload=500698]: Got next segment. Inline: false Dec 29 10:36:02.517125212 (571h7m2.388208267s) [upload=500698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:36:02.517130832 (571h7m2.388213877s) [upload=500698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:36:02.765279145 (571h7m2.63636219s) [seg_pos={0 0}, upload=500698]: Begin upload segment... Dec 29 10:36:02.765338348 (571h7m2.636421393s) [seg_pos={0 0}, upload=500698]: Done begin upload segment. Dec 29 10:36:02.765374908 (571h7m2.636457953s) [noise=true , address=145.79.227.14:4839 , piece_id=N4U5MSRE3MSWRIOYKGLPHDXJMOWHXU6XC2MQTI53PZCN23TP2LTA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.765523049 (571h7m2.636606084s) [noise=true , address=145.79.227.38:4977 , piece_id=3QAS32F2RSMIQFAEAVANSP5MCVNAYGJTNDW7ZCGDDIORCOGLTBKA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.766436089 (571h7m2.637519134s) [noise=true , address=77.90.31.12:4589 , piece_id=MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.766943835 (571h7m2.63802689s) [noise=true , address=77.90.31.16:4735 , piece_id=J4NKX4S3WEQZ75P2QAGNHIV76PG7BKHEW7K5IJUPHVIIOCN6WVYA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.767456161 (571h7m2.638539206s) [noise=true , address=45.91.116.8:4047 , piece_id=OFQDGRAZOUBHEZ7JOQCVAZMIZPCGPKZPCNAB7WXDAI44NZASWO7Q, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.767917749 (571h7m2.639000795s) [noise=true , address=45.91.116.7:4010 , piece_id=7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.768387594 (571h7m2.639470639s) [noise=true , address=145.79.227.20:4437 , piece_id=YCOJURIYPT7A736I7RBYZOWYDISPVDPUZQ2P5Q5FEHWPUCFQUHLA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.768874020 (571h7m2.639957065s) [noise=true , address=145.79.227.7:4293 , piece_id=FGTHHPPEY4SJYAJYGVV6P6U6LOXVEIYGE7COZ2M7IUI73QJTRZMQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.770931558 (571h7m2.642014613s) [noise=true , address=145.79.227.39:4013 , piece_id=VMK54RDXXLRO5JFPHXY65XOTQGFPISLMGNU4GAHFUUCIVRHTY5IA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.771934588 (571h7m2.643017633s) [noise=true , address=145.79.227.6:4259 , piece_id=QSQWMG2EQMIRFSDK2B6ODOLYBTYPCNSZAXK6DP2L7F5VPHBBN3IQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.773055302 (571h7m2.644138347s) [noise=true , address=77.90.31.13:4629 , piece_id=H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.775107159 (571h7m2.646190214s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.776727263 (571h7m2.647810308s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.778602304 (571h7m2.649685339s) [noise=true , address=145.79.227.45:4229 , piece_id=4PWMCZW7I7IONT56GD43XWH4OOYXGLCFEPWKEXA5DPBSIQGLMMPQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.780232248 (571h7m2.651315293s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.788867992 (571h7m2.659951037s) [noise=true , address=145.79.227.19:4078 , piece_id=5QG2HNCMSQ7NGWLZ736KWLOVP5UP3NSVWOUYBV6J2COYOQ4U4FTQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:04.377125647 (571h7m4.248208682s) [upload=500698]: Next returned nil segment Dec 29 10:36:04.377141577 (571h7m4.248224642s) [upload=500698]: Waiting for error group managing segments... Dec 29 10:36:04.757104383 (571h7m4.628187428s) [noise=true , address=145.79.227.20:4437 , piece_id=YCOJURIYPT7A736I7RBYZOWYDISPVDPUZQ2P5Q5FEHWPUCFQUHLA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.762309806 (571h7m4.633392851s) [noise=true , address=145.79.227.45:4229 , piece_id=4PWMCZW7I7IONT56GD43XWH4OOYXGLCFEPWKEXA5DPBSIQGLMMPQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.764708964 (571h7m4.635792009s) [noise=true , address=145.79.227.38:4977 , piece_id=3QAS32F2RSMIQFAEAVANSP5MCVNAYGJTNDW7ZCGDDIORCOGLTBKA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.765123674 (571h7m4.636206709s) [noise=true , address=145.79.227.6:4259 , piece_id=QSQWMG2EQMIRFSDK2B6ODOLYBTYPCNSZAXK6DP2L7F5VPHBBN3IQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.771422879 (571h7m4.642505924s) [noise=true , address=145.79.227.14:4839 , piece_id=N4U5MSRE3MSWRIOYKGLPHDXJMOWHXU6XC2MQTI53PZCN23TP2LTA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.773345269 (571h7m4.644428314s) [noise=true , address=145.79.227.19:4078 , piece_id=5QG2HNCMSQ7NGWLZ736KWLOVP5UP3NSVWOUYBV6J2COYOQ4U4FTQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.773381989 (571h7m4.644465034s) [noise=true , address=145.79.227.39:4013 , piece_id=VMK54RDXXLRO5JFPHXY65XOTQGFPISLMGNU4GAHFUUCIVRHTY5IA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.787804893 (571h7m4.658887938s) [noise=true , address=145.79.227.7:4293 , piece_id=FGTHHPPEY4SJYAJYGVV6P6U6LOXVEIYGE7COZ2M7IUI73QJTRZMQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.022385700 (571h7m4.893468745s) [noise=true , address=45.91.116.7:4010 , piece_id=7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.036273144 (571h7m4.907356189s) [noise=true , address=77.90.31.13:4629 , piece_id=H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.038862995 (571h7m4.909946041s) [noise=true , address=45.91.116.8:4047 , piece_id=OFQDGRAZOUBHEZ7JOQCVAZMIZPCGPKZPCNAB7WXDAI44NZASWO7Q, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.108104462 (571h7m4.979187507s) [noise=true , address=77.90.31.16:4735 , piece_id=J4NKX4S3WEQZ75P2QAGNHIV76PG7BKHEW7K5IJUPHVIIOCN6WVYA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.155902817 (571h7m5.026985862s) [noise=true , address=77.90.31.12:4589 , piece_id=MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.155931702 (571h7m5.027014748s) [seg_pos={0 0}, upload=500698]: Segment reached optimal threshold of 13 pieces. Dec 29 10:36:05.156099362 (571h7m5.027182407s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:05.156106034 (571h7m5.027189079s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156382421 (571h7m5.027465496s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection Dec 29 10:36:05.156395575 (571h7m5.027478651s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156547274 (571h7m5.027630309s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:05.156553787 (571h7m5.027636822s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156571721 (571h7m5.027654756s) [seg_pos={0 0}, upload=500698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:36:05.156634861 (571h7m5.027717906s) [upload=500698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:05.156639560 (571h7m5.027722605s) [upload=500698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:05.444302965 (571h7m5.315386s) [upload=500698]: Done uploading segments... Dec 29 10:36:05.444323063 (571h7m5.315406098s) [upload=500698]: Upload finished. err: <nil> Duration: 5.740213043s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..................X.......................................X] duration: 1m0.003256755s trace: https://jaeger.dev.storj.io/trace/29c8b9abfc97977f FAIL (2025-12-29 10:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding piece... Dec 29 10:36:09.625240268 (571h7m9.496323313s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.626432970 (571h7m9.497516015s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627296334 (571h7m9.498379369s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.627911675 (571h7m9.49899472s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628402149 (571h7m9.499485184s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.628746605 (571h7m9.49982964s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629115037 (571h7m9.500198082s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629529396 (571h7m9.500612441s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.629961909 (571h7m9.501044944s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630419400 (571h7m9.501502445s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.630971271 (571h7m9.502054306s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631417361 (571h7m9.502500406s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.631932682 (571h7m9.503015717s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632402456 (571h7m9.503485491s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.632843356 (571h7m9.503926391s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:09.633041223 (571h7m9.504124278s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Uploading piece... Dec 29 10:36:10.920134064 (571h7m10.791217099s) [upload=500702]: Next returned nil segment Dec 29 10:36:10.920146147 (571h7m10.791229182s) [upload=500702]: Waiting for error group managing segments... Dec 29 10:36:11.250718097 (571h7m11.121801132s) [noise=true , address=64.49.13.26:4948 , piece_id=BWOLS4GEECYQIUZBHWJGMEBIC2HWATNDFMSGRDZRZNDZIN344IZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.355970684 (571h7m11.227053729s) [noise=true , address=64.49.13.155:4913 , piece_id=FDDQKQ3RYKL6GO6T2F7MAQH2YVBYKSYZZF6QBRWHNGRKDD5W24RQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.480191450 (571h7m11.351274485s) [noise=true , address=145.79.227.46:4624 , piece_id=URPDZYR7KLJMHGI6ZVQOXAJUR2D2G4HIEJC2NA7JZPG2EOW2SVZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.481450216 (571h7m11.352533251s) [noise=true , address=145.79.227.51:4431 , piece_id=6QXLHOWTL2KNCS7UJ3GZTI3Y6PMHPNIRSJEILIQ5FAGGCLOOKBJA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.517631081 (571h7m11.388714116s) [noise=true , address=145.79.227.17:4984 , piece_id=PMT6M2PMKZFQN6K2VY4R6WBWCFHQKRYNSRHMJUPH6EUODPX6GOOA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.532220150 (571h7m11.403303195s) [noise=true , address=145.79.227.40:4068 , piece_id=QC77FUGHTRE3IMY7LEJTZEPUYXVCCK72FMKURDRIC2ANJAJHFUGA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.732386658 (571h7m11.603469703s) [noise=true , address=45.91.116.5:4304 , piece_id=Y3IEF3ZAGWQUD6GINFUGK44QLGFE5FCZXVVZEPVG2BKRJNU3AVDA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.782029176 (571h7m11.653112211s) [noise=true , address=77.90.31.11:4585 , piece_id=QGOHFDR4NFZVDEAETPJ5E3L456N3V3LKEFJPI7QGKLVALCBTF5NQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:11.900912678 (571h7m11.771995723s) [noise=true , address=77.90.31.17:4769 , piece_id=EIMGE5CTIFPJNMYWY2FQSFATC7S4OIGSTAFMD2YVQGB7JQO3AMZQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.222186744 (571h7m12.093269779s) [noise=true , address=193.149.164.143:4701 , piece_id=OZRCIQKEBYOUZKUTCGHKGSGD574KFGQB7ZCVXIUGKH3EQLPL6P2Q, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.423657333 (571h7m12.294740368s) [noise=true , address=145.79.227.30:4770 , piece_id=WZ2NMRKLSNM4GX52DGJSHCJMRWBCCWK7GGANHEB3K4ZVKHQEJ55A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.452443096 (571h7m12.323526131s) [noise=true , address=145.79.227.29:4946 , piece_id=QKHT6H4DBA743WDZ3TNYL2PPX25FSZ7NSZNCYSUHWVMISIELZ5SQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.480983001 (571h7m12.352066036s) [noise=true , address=145.79.227.49:4352 , piece_id=WRTSYBYELQRJSLHCRVLLKVXUZE7ABFOH7D4Q27HVEQFHOMBWVGSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481960592 (571h7m12.353043637s) [noise=true , address=145.79.227.44:4210 , piece_id=4OG4DNIQ2ODAZZI6YEPU4PYZED5RINROA5DF2XHWYB66MU5HIBSA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: <nil> Dec 29 10:36:12.481971863 (571h7m12.353054899s) [seg_pos={0 0}, upload=500702]: Segment reached optimal threshold of 14 pieces. Dec 29 10:36:12.482148199 (571h7m12.353231234s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32950->64.49.13.169:4707: use of closed network connection Dec 29 10:36:12.482156334 (571h7m12.35323938s) [noise=true , address=64.49.13.169:4707 , piece_id=VLESNOAWJIMQ7L27BLFONOYS5R3G72WUXNVMQUL2OUQEAVPJWZYQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482224615 (571h7m12.35330766s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482234202 (571h7m12.353317238s) [noise=true , address=145.79.227.33:4885 , piece_id=EXN6XCHWTK36LLLBYKUIGC4HV7DRJHC4FDAWKW6DNZATMDHTS7QA, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482297102 (571h7m12.353380148s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482302823 (571h7m12.353385858s) [noise=true , address=145.79.227.34:4918 , piece_id=WARLAFG3RHIKUDGEYDFXCVDYE4PW55COT2RPU6BU5HUQJBB42N5A, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482418053 (571h7m12.353501088s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:12.482423514 (571h7m12.353506559s) [noise=true , address=77.90.31.18:4818 , piece_id=WOE5PSEKQYVUTFP6RSVWPDVOG6SPJKM357WU5TZ63QEQNA7UXNKQ, seg_pos={0 0}, upload=500702]: Long tail cancellation due to optimal threshold Dec 29 10:36:12.482498005 (571h7m12.35358105s) [seg_pos={0 0}, upload=500702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:36:12.482565885 (571h7m12.35364893s) [upload=500702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:12.482570483 (571h7m12.353653518s) [upload=500702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:12.818180257 (571h7m12.689263302s) [upload=500702]: Done uploading segments... Dec 29 10:36:12.818217258 (571h7m12.689300303s) [upload=500702]: Upload finished. err: <nil> Duration: 1m0.003270922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX..XXX......X..X.......X....X..............X...X.X.X....X..] duration: 3.418720057s trace: https://jaeger.dev.storj.io/trace/60335edd44b6c64f FAIL (2025-12-29 10:36:39 UTC) orbiter warning: context deadline exceeded Duration: 3.418747919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X..X....X...X..XX...X..XX..X......X.X.X..X..........X...X] duration: 30.003148813s trace: https://jaeger.dev.storj.io/trace/bd2bae9b250254b FAIL (2025-12-29 10:36:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA) context deadline exceeded ======LOGS====== Dec 29 10:36:02.516563031 (571h7m2.387646076s) [upload=500698]: Starting upload Dec 29 10:36:02.516647832 (571h7m2.387730877s) [upload=500698]: Done starting upload Dec 29 10:36:02.516725099 (571h7m2.387808144s) [upload=500698]: Uploading segments... Dec 29 10:36:02.517074164 (571h7m2.388157219s) [upload=500698]: Got next segment. Inline: false Dec 29 10:36:02.517125212 (571h7m2.388208267s) [upload=500698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:36:02.517130832 (571h7m2.388213877s) [upload=500698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:36:02.765279145 (571h7m2.63636219s) [seg_pos={0 0}, upload=500698]: Begin upload segment... Dec 29 10:36:02.765338348 (571h7m2.636421393s) [seg_pos={0 0}, upload=500698]: Done begin upload segment. Dec 29 10:36:02.765374908 (571h7m2.636457953s) [noise=true , address=145.79.227.14:4839 , piece_id=N4U5MSRE3MSWRIOYKGLPHDXJMOWHXU6XC2MQTI53PZCN23TP2LTA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.765523049 (571h7m2.636606084s) [noise=true , address=145.79.227.38:4977 , piece_id=3QAS32F2RSMIQFAEAVANSP5MCVNAYGJTNDW7ZCGDDIORCOGLTBKA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.766436089 (571h7m2.637519134s) [noise=true , address=77.90.31.12:4589 , piece_id=MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.766943835 (571h7m2.63802689s) [noise=true , address=77.90.31.16:4735 , piece_id=J4NKX4S3WEQZ75P2QAGNHIV76PG7BKHEW7K5IJUPHVIIOCN6WVYA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.767456161 (571h7m2.638539206s) [noise=true , address=45.91.116.8:4047 , piece_id=OFQDGRAZOUBHEZ7JOQCVAZMIZPCGPKZPCNAB7WXDAI44NZASWO7Q, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.767917749 (571h7m2.639000795s) [noise=true , address=45.91.116.7:4010 , piece_id=7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.768387594 (571h7m2.639470639s) [noise=true , address=145.79.227.20:4437 , piece_id=YCOJURIYPT7A736I7RBYZOWYDISPVDPUZQ2P5Q5FEHWPUCFQUHLA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.768874020 (571h7m2.639957065s) [noise=true , address=145.79.227.7:4293 , piece_id=FGTHHPPEY4SJYAJYGVV6P6U6LOXVEIYGE7COZ2M7IUI73QJTRZMQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.770931558 (571h7m2.642014613s) [noise=true , address=145.79.227.39:4013 , piece_id=VMK54RDXXLRO5JFPHXY65XOTQGFPISLMGNU4GAHFUUCIVRHTY5IA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.771934588 (571h7m2.643017633s) [noise=true , address=145.79.227.6:4259 , piece_id=QSQWMG2EQMIRFSDK2B6ODOLYBTYPCNSZAXK6DP2L7F5VPHBBN3IQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.773055302 (571h7m2.644138347s) [noise=true , address=77.90.31.13:4629 , piece_id=H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.775107159 (571h7m2.646190214s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.776727263 (571h7m2.647810308s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.778602304 (571h7m2.649685339s) [noise=true , address=145.79.227.45:4229 , piece_id=4PWMCZW7I7IONT56GD43XWH4OOYXGLCFEPWKEXA5DPBSIQGLMMPQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.780232248 (571h7m2.651315293s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:02.788867992 (571h7m2.659951037s) [noise=true , address=145.79.227.19:4078 , piece_id=5QG2HNCMSQ7NGWLZ736KWLOVP5UP3NSVWOUYBV6J2COYOQ4U4FTQ, seg_pos={0 0}, upload=500698]: Uploading piece... Dec 29 10:36:04.377125647 (571h7m4.248208682s) [upload=500698]: Next returned nil segment Dec 29 10:36:04.377141577 (571h7m4.248224642s) [upload=500698]: Waiting for error group managing segments... Dec 29 10:36:04.757104383 (571h7m4.628187428s) [noise=true , address=145.79.227.20:4437 , piece_id=YCOJURIYPT7A736I7RBYZOWYDISPVDPUZQ2P5Q5FEHWPUCFQUHLA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.762309806 (571h7m4.633392851s) [noise=true , address=145.79.227.45:4229 , piece_id=4PWMCZW7I7IONT56GD43XWH4OOYXGLCFEPWKEXA5DPBSIQGLMMPQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.764708964 (571h7m4.635792009s) [noise=true , address=145.79.227.38:4977 , piece_id=3QAS32F2RSMIQFAEAVANSP5MCVNAYGJTNDW7ZCGDDIORCOGLTBKA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.765123674 (571h7m4.636206709s) [noise=true , address=145.79.227.6:4259 , piece_id=QSQWMG2EQMIRFSDK2B6ODOLYBTYPCNSZAXK6DP2L7F5VPHBBN3IQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.771422879 (571h7m4.642505924s) [noise=true , address=145.79.227.14:4839 , piece_id=N4U5MSRE3MSWRIOYKGLPHDXJMOWHXU6XC2MQTI53PZCN23TP2LTA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.773345269 (571h7m4.644428314s) [noise=true , address=145.79.227.19:4078 , piece_id=5QG2HNCMSQ7NGWLZ736KWLOVP5UP3NSVWOUYBV6J2COYOQ4U4FTQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.773381989 (571h7m4.644465034s) [noise=true , address=145.79.227.39:4013 , piece_id=VMK54RDXXLRO5JFPHXY65XOTQGFPISLMGNU4GAHFUUCIVRHTY5IA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:04.787804893 (571h7m4.658887938s) [noise=true , address=145.79.227.7:4293 , piece_id=FGTHHPPEY4SJYAJYGVV6P6U6LOXVEIYGE7COZ2M7IUI73QJTRZMQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.022385700 (571h7m4.893468745s) [noise=true , address=45.91.116.7:4010 , piece_id=7MMNRGNCEF2IWFCZNIPLMERH5TJVPPXRUNRUO24HROCVETJP7WJA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.036273144 (571h7m4.907356189s) [noise=true , address=77.90.31.13:4629 , piece_id=H7KMSARPEARXDQPRL7YPT5KVYLFJXG64O4VPHAPM2K2BR6T5RSQA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.038862995 (571h7m4.909946041s) [noise=true , address=45.91.116.8:4047 , piece_id=OFQDGRAZOUBHEZ7JOQCVAZMIZPCGPKZPCNAB7WXDAI44NZASWO7Q, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.108104462 (571h7m4.979187507s) [noise=true , address=77.90.31.16:4735 , piece_id=J4NKX4S3WEQZ75P2QAGNHIV76PG7BKHEW7K5IJUPHVIIOCN6WVYA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.155902817 (571h7m5.026985862s) [noise=true , address=77.90.31.12:4589 , piece_id=MASRA4NJ7UHFBLXPMPEIYXJCEZV4GIQC2OOWWVU5J43A4G3BGMXQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: <nil> Dec 29 10:36:05.155931702 (571h7m5.027014748s) [seg_pos={0 0}, upload=500698]: Segment reached optimal threshold of 13 pieces. Dec 29 10:36:05.156099362 (571h7m5.027182407s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:05.156106034 (571h7m5.027189079s) [noise=true , address=45.91.116.4:4937 , piece_id=4IPVAX27Z4EA5MHQ5EH37XHSYIPQKMPJZLAD5HY46Z3MJQNMUWEA, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156382421 (571h7m5.027465496s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44448->45.91.116.12:4228: use of closed network connection Dec 29 10:36:05.156395575 (571h7m5.027478651s) [noise=true , address=45.91.116.12:4228 , piece_id=LW5DNYW7SB7SQBGRJXFW5KWNCO7X6EAYIYTKEQEGNPAPLBXDDSUQ, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156547274 (571h7m5.027630309s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 10:36:05.156553787 (571h7m5.027636822s) [noise=true , address=77.90.31.15:4697 , piece_id=MLUVRIO63F3AE5BF3JKMDHUPGD355R624VEE3QHGOJ62LL2E7FOQ, seg_pos={0 0}, upload=500698]: Long tail cancellation due to optimal threshold Dec 29 10:36:05.156571721 (571h7m5.027654756s) [seg_pos={0 0}, upload=500698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:36:05.156634861 (571h7m5.027717906s) [upload=500698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:36:05.156639560 (571h7m5.027722605s) [upload=500698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:36:05.444302965 (571h7m5.315386s) [upload=500698]: Done uploading segments... Dec 29 10:36:05.444323063 (571h7m5.315406098s) [upload=500698]: Upload finished. err: <nil> Duration: 30.003172928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..................X.......................................] duration: 22.935134897s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 22.935166676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...X..X....X...X..XX...X..XX..X......X.X.X..X..........X...] duration: 5.988729327s trace: https://jaeger.dev.storj.io/trace/1d73de7d77af0f03 FAIL (2025-12-29 10:35:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:32:28.620422388 (571h3m28.491505423s) [upload=500655]: Starting upload Dec 29 10:32:28.620485138 (571h3m28.491568173s) [upload=500655]: Done starting upload Dec 29 10:32:28.620574558 (571h3m28.491657593s) [upload=500655]: Uploading segments... Dec 29 10:32:28.620684517 (571h3m28.491767552s) [upload=500655]: Got next segment. Inline: false Dec 29 10:32:28.620723431 (571h3m28.491806466s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:32:28.620728541 (571h3m28.491811586s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:32:28.864323894 (571h3m28.735406949s) [seg_pos={0 0}, upload=500655]: Begin upload segment... Dec 29 10:32:28.864380152 (571h3m28.735463187s) [seg_pos={0 0}, upload=500655]: Done begin upload segment. Dec 29 10:32:28.864398085 (571h3m28.73548113s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864412173 (571h3m28.735495218s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864601473 (571h3m28.735684528s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865153855 (571h3m28.73623692s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865679896 (571h3m28.736762941s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866243650 (571h3m28.737326695s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866663118 (571h3m28.737746163s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867067268 (571h3m28.738150313s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867527635 (571h3m28.73861068s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868009472 (571h3m28.739092517s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868498123 (571h3m28.739581158s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869685494 (571h3m28.740768539s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869994121 (571h3m28.741077167s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.871545695 (571h3m28.74262873s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.873107550 (571h3m28.744190605s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.874634227 (571h3m28.745717282s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:30.486480128 (571h3m30.357563173s) [upload=500655]: Next returned nil segment Dec 29 10:32:30.486495608 (571h3m30.357578653s) [upload=500655]: Waiting for error group managing segments... Dec 29 10:32:30.864769148 (571h3m30.735852183s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865551849 (571h3m30.736634894s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865968362 (571h3m30.737051397s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.875937754 (571h3m30.747020799s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.880115028 (571h3m30.751198064s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.883694364 (571h3m30.754777399s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.887159563 (571h3m30.758242608s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.889914609 (571h3m30.760997654s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.899663252 (571h3m30.770746287s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.073060677 (571h3m30.944143712s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.643736261 (571h3m31.514819306s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.735097406 (571h3m31.606180441s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218081181 (571h3m32.089164216s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218111699 (571h3m32.089194734s) [seg_pos={0 0}, upload=500655]: Segment reached optimal threshold of 13 pieces. Dec 29 10:32:32.218275912 (571h3m32.089358947s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218284017 (571h3m32.089367052s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218500810 (571h3m32.089583845s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection Dec 29 10:32:32.218518975 (571h3m32.08960203s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218540695 (571h3m32.089623731s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218548040 (571h3m32.089631085s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218621149 (571h3m32.089704194s) [seg_pos={0 0}, upload=500655]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:32:32.218725227 (571h3m32.089808263s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:32:32.218730517 (571h3m32.089813562s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:32:32.505652691 (571h3m32.376735736s) [upload=500655]: Done uploading segments... Dec 29 10:32:32.505680754 (571h3m32.376763789s) [upload=500655]: Upload finished. err: <nil> Duration: 5.988756148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXX..XXX......X..X.......X....X..............X...X.X.X....X.] duration: 3.785545915s trace: https://jaeger.dev.storj.io/trace/60335edd44b6c64f FAIL (2025-12-29 10:35:30 UTC) orbiter warning: context deadline exceeded Duration: 3.785610449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XXX..XXX......X..X.......X....X..............X...X.X.X....X] duration: 5.000589057s trace: https://jaeger.dev.storj.io/trace/60335edd44b6c64f FAIL (2025-12-29 10:35:07 UTC) orbiter warning: context deadline exceeded Duration: 5.000610878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..................X......................................] duration: 27.550612338s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:34:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 27.550630223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..XXX..XXX......X..X.......X....X..............X...X.X.X....] duration: 5.001977531s trace: https://jaeger.dev.storj.io/trace/7b698b41d2122118 FAIL (2025-12-29 10:34:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002003891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X...X..X....X...X..XX...X..XX..X......X.X.X..X..........X..] duration: 5.926836773s trace: https://jaeger.dev.storj.io/trace/1d73de7d77af0f03 FAIL (2025-12-29 10:34:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:32:28.620422388 (571h3m28.491505423s) [upload=500655]: Starting upload Dec 29 10:32:28.620485138 (571h3m28.491568173s) [upload=500655]: Done starting upload Dec 29 10:32:28.620574558 (571h3m28.491657593s) [upload=500655]: Uploading segments... Dec 29 10:32:28.620684517 (571h3m28.491767552s) [upload=500655]: Got next segment. Inline: false Dec 29 10:32:28.620723431 (571h3m28.491806466s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:32:28.620728541 (571h3m28.491811586s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:32:28.864323894 (571h3m28.735406949s) [seg_pos={0 0}, upload=500655]: Begin upload segment... Dec 29 10:32:28.864380152 (571h3m28.735463187s) [seg_pos={0 0}, upload=500655]: Done begin upload segment. Dec 29 10:32:28.864398085 (571h3m28.73548113s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864412173 (571h3m28.735495218s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864601473 (571h3m28.735684528s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865153855 (571h3m28.73623692s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865679896 (571h3m28.736762941s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866243650 (571h3m28.737326695s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866663118 (571h3m28.737746163s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867067268 (571h3m28.738150313s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867527635 (571h3m28.73861068s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868009472 (571h3m28.739092517s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868498123 (571h3m28.739581158s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869685494 (571h3m28.740768539s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869994121 (571h3m28.741077167s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.871545695 (571h3m28.74262873s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.873107550 (571h3m28.744190605s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.874634227 (571h3m28.745717282s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:30.486480128 (571h3m30.357563173s) [upload=500655]: Next returned nil segment Dec 29 10:32:30.486495608 (571h3m30.357578653s) [upload=500655]: Waiting for error group managing segments... Dec 29 10:32:30.864769148 (571h3m30.735852183s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865551849 (571h3m30.736634894s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865968362 (571h3m30.737051397s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.875937754 (571h3m30.747020799s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.880115028 (571h3m30.751198064s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.883694364 (571h3m30.754777399s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.887159563 (571h3m30.758242608s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.889914609 (571h3m30.760997654s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.899663252 (571h3m30.770746287s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.073060677 (571h3m30.944143712s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.643736261 (571h3m31.514819306s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.735097406 (571h3m31.606180441s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218081181 (571h3m32.089164216s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218111699 (571h3m32.089194734s) [seg_pos={0 0}, upload=500655]: Segment reached optimal threshold of 13 pieces. Dec 29 10:32:32.218275912 (571h3m32.089358947s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218284017 (571h3m32.089367052s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218500810 (571h3m32.089583845s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection Dec 29 10:32:32.218518975 (571h3m32.08960203s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218540695 (571h3m32.089623731s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218548040 (571h3m32.089631085s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218621149 (571h3m32.089704194s) [seg_pos={0 0}, upload=500655]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:32:32.218725227 (571h3m32.089808263s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:32:32.218730517 (571h3m32.089813562s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:32:32.505652691 (571h3m32.376735736s) [upload=500655]: Done uploading segments... Dec 29 10:32:32.505680754 (571h3m32.376763789s) [upload=500655]: Upload finished. err: <nil> Duration: 5.926864086s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...XXX..XXX......X..X.......X....X..............X...X.X.X...] duration: 4.514356052s trace: https://jaeger.dev.storj.io/trace/7b698b41d2122118 FAIL (2025-12-29 10:33:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.514381951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..................X.....................................] duration: 20.751080652s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 20.751114486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....XXX..XXX......X..X.......X....X..............X...X.X.X..] duration: 5.001405133s trace: https://jaeger.dev.storj.io/trace/7b698b41d2122118 FAIL (2025-12-29 10:33:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001429471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X...X..X....X...X..XX...X..XX..X......X.X.X..X..........X.] duration: 5.930748045s trace: https://jaeger.dev.storj.io/trace/1d73de7d77af0f03 FAIL (2025-12-29 10:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:32:28.620422388 (571h3m28.491505423s) [upload=500655]: Starting upload Dec 29 10:32:28.620485138 (571h3m28.491568173s) [upload=500655]: Done starting upload Dec 29 10:32:28.620574558 (571h3m28.491657593s) [upload=500655]: Uploading segments... Dec 29 10:32:28.620684517 (571h3m28.491767552s) [upload=500655]: Got next segment. Inline: false Dec 29 10:32:28.620723431 (571h3m28.491806466s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:32:28.620728541 (571h3m28.491811586s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:32:28.864323894 (571h3m28.735406949s) [seg_pos={0 0}, upload=500655]: Begin upload segment... Dec 29 10:32:28.864380152 (571h3m28.735463187s) [seg_pos={0 0}, upload=500655]: Done begin upload segment. Dec 29 10:32:28.864398085 (571h3m28.73548113s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864412173 (571h3m28.735495218s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864601473 (571h3m28.735684528s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865153855 (571h3m28.73623692s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865679896 (571h3m28.736762941s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866243650 (571h3m28.737326695s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866663118 (571h3m28.737746163s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867067268 (571h3m28.738150313s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867527635 (571h3m28.73861068s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868009472 (571h3m28.739092517s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868498123 (571h3m28.739581158s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869685494 (571h3m28.740768539s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869994121 (571h3m28.741077167s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.871545695 (571h3m28.74262873s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.873107550 (571h3m28.744190605s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.874634227 (571h3m28.745717282s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:30.486480128 (571h3m30.357563173s) [upload=500655]: Next returned nil segment Dec 29 10:32:30.486495608 (571h3m30.357578653s) [upload=500655]: Waiting for error group managing segments... Dec 29 10:32:30.864769148 (571h3m30.735852183s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865551849 (571h3m30.736634894s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865968362 (571h3m30.737051397s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.875937754 (571h3m30.747020799s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.880115028 (571h3m30.751198064s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.883694364 (571h3m30.754777399s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.887159563 (571h3m30.758242608s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.889914609 (571h3m30.760997654s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.899663252 (571h3m30.770746287s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.073060677 (571h3m30.944143712s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.643736261 (571h3m31.514819306s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.735097406 (571h3m31.606180441s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218081181 (571h3m32.089164216s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218111699 (571h3m32.089194734s) [seg_pos={0 0}, upload=500655]: Segment reached optimal threshold of 13 pieces. Dec 29 10:32:32.218275912 (571h3m32.089358947s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218284017 (571h3m32.089367052s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218500810 (571h3m32.089583845s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection Dec 29 10:32:32.218518975 (571h3m32.08960203s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218540695 (571h3m32.089623731s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218548040 (571h3m32.089631085s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218621149 (571h3m32.089704194s) [seg_pos={0 0}, upload=500655]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:32:32.218725227 (571h3m32.089808263s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:32:32.218730517 (571h3m32.089813562s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:32:32.505652691 (571h3m32.376735736s) [upload=500655]: Done uploading segments... Dec 29 10:32:32.505680754 (571h3m32.376763789s) [upload=500655]: Upload finished. err: <nil> Duration: 5.930778112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..................X....................................] duration: 44.327228303s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 44.327245786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X...X..X....X...X..XX...X..XX..X......X.X.X..X..........X] duration: 30.002785031s trace: https://jaeger.dev.storj.io/trace/1d73de7d77af0f03 FAIL (2025-12-29 10:32:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:32:28.620422388 (571h3m28.491505423s) [upload=500655]: Starting upload Dec 29 10:32:28.620485138 (571h3m28.491568173s) [upload=500655]: Done starting upload Dec 29 10:32:28.620574558 (571h3m28.491657593s) [upload=500655]: Uploading segments... Dec 29 10:32:28.620684517 (571h3m28.491767552s) [upload=500655]: Got next segment. Inline: false Dec 29 10:32:28.620723431 (571h3m28.491806466s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:32:28.620728541 (571h3m28.491811586s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:32:28.864323894 (571h3m28.735406949s) [seg_pos={0 0}, upload=500655]: Begin upload segment... Dec 29 10:32:28.864380152 (571h3m28.735463187s) [seg_pos={0 0}, upload=500655]: Done begin upload segment. Dec 29 10:32:28.864398085 (571h3m28.73548113s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864412173 (571h3m28.735495218s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.864601473 (571h3m28.735684528s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865153855 (571h3m28.73623692s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.865679896 (571h3m28.736762941s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866243650 (571h3m28.737326695s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.866663118 (571h3m28.737746163s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867067268 (571h3m28.738150313s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.867527635 (571h3m28.73861068s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868009472 (571h3m28.739092517s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.868498123 (571h3m28.739581158s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869685494 (571h3m28.740768539s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.869994121 (571h3m28.741077167s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.871545695 (571h3m28.74262873s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.873107550 (571h3m28.744190605s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:28.874634227 (571h3m28.745717282s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Uploading piece... Dec 29 10:32:30.486480128 (571h3m30.357563173s) [upload=500655]: Next returned nil segment Dec 29 10:32:30.486495608 (571h3m30.357578653s) [upload=500655]: Waiting for error group managing segments... Dec 29 10:32:30.864769148 (571h3m30.735852183s) [noise=true , address=145.79.227.37:4565 , piece_id=IKW5FLJV2MWUTAX7WYE4UT6JFDKVAIWTX6LTRBI6VH2CHP7IBWNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865551849 (571h3m30.736634894s) [noise=true , address=145.79.227.26:4654 , piece_id=I22BYP5ZD3Q3FA45U234ORNHQQHWDFAFQ2PK5H6LKOGXULILUGJA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.865968362 (571h3m30.737051397s) [noise=true , address=145.79.227.19:4077 , piece_id=AAHTD62ECB5HMMCC47OCBBBCVNURLDWMLS75LMEDCIE76ULNNFNA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.875937754 (571h3m30.747020799s) [noise=true , address=145.79.227.14:4837 , piece_id=B46YQOGVJCDOUVPYGEO4JVWF6EFNXRP7RIYOGCJT7OGPSSOWJN7Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.880115028 (571h3m30.751198064s) [noise=true , address=145.79.227.11:4730 , piece_id=QGB3AOE5S74A6TGQJJYAZKE5UWM4Q4LLBXAOHX7IJKWVYXZL2S6Q, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.883694364 (571h3m30.754777399s) [noise=true , address=145.79.227.15:4875 , piece_id=RRRKVG7HQSX763TXUXT2I5K7NVT2MKXRHJTBNOBD6UZK6EVRI43A, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.887159563 (571h3m30.758242608s) [noise=true , address=145.79.227.51:4411 , piece_id=KZPZBDLFYZCMHT7GPJ3LBKBTAZ2OCT45TKPALXU74N3DAIBOS5FA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.889914609 (571h3m30.760997654s) [noise=true , address=145.79.227.48:4301 , piece_id=TLYW7RQ4YHENP74ZKEO4RBEOLHXTLRRC2ADFLKTP74G6T3DOEPSQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:30.899663252 (571h3m30.770746287s) [noise=true , address=145.79.227.47:4266 , piece_id=2IOKVCFY4QOFTHQ2O3USJHIQUGZ6LAYJGYE4SB4SF4XWRYNNXCBA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.073060677 (571h3m30.944143712s) [noise=true , address=77.90.31.14:4662 , piece_id=2YSFY43JHB6LKIXRPYKGOZUKDW3O3Z2QLNVFQVUH45JJPYDEQXQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.643736261 (571h3m31.514819306s) [noise=true , address=45.91.116.9:4264 , piece_id=JMZHHB6KOPO2FKCYSA6EFYHGI6PNCH3Q47KEWI5RYIA56DLN3FTA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:31.735097406 (571h3m31.606180441s) [noise=true , address=77.90.31.17:4770 , piece_id=2URS4XGHIE4QONADLDBZ5TAFPVRDL5N6BTHSVVKHFH3WQR2RLJZA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218081181 (571h3m32.089164216s) [noise=true , address=45.91.116.5:4297 , piece_id=7JT4OB5BLOACY55DSZNGVWKLU44VJRIETIKZ43ERYHAHFV6GEDQA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: <nil> Dec 29 10:32:32.218111699 (571h3m32.089194734s) [seg_pos={0 0}, upload=500655]: Segment reached optimal threshold of 13 pieces. Dec 29 10:32:32.218275912 (571h3m32.089358947s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218284017 (571h3m32.089367052s) [noise=true , address=145.79.227.43:4158 , piece_id=JP2XMX5MU56472GKC4NKLHZATNOHQEOS4SJ7C3VKCJBI63SEHPAQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218500810 (571h3m32.089583845s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59388->45.91.116.10:4084: use of closed network connection Dec 29 10:32:32.218518975 (571h3m32.08960203s) [noise=true , address=45.91.116.10:4084 , piece_id=IUDOQ63A6VX7LVIUHHAC5BDV2Z7264X3P2JV2XWPUTBXXWDLQXYA, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218540695 (571h3m32.089623731s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:32:32.218548040 (571h3m32.089631085s) [noise=true , address=145.79.227.49:4337 , piece_id=4JQGPTW2MFQTKWKH42JUFX2YSL53DOUKC2UQCQGKWAL7SIEGO3QQ, seg_pos={0 0}, upload=500655]: Long tail cancellation due to optimal threshold Dec 29 10:32:32.218621149 (571h3m32.089704194s) [seg_pos={0 0}, upload=500655]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:32:32.218725227 (571h3m32.089808263s) [upload=500655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:32:32.218730517 (571h3m32.089813562s) [upload=500655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:32:32.505652691 (571h3m32.376735736s) [upload=500655]: Done uploading segments... Dec 29 10:32:32.505680754 (571h3m32.376763789s) [upload=500655]: Upload finished. err: <nil> Duration: 30.002812283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..................X...................................] duration: 38.761356078s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 38.761383551s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)