Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.626037251s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:18:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.62606788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.........X.....................] duration: 6.013543781s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 6.013572546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.547390384s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:17:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54741476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.........X....................] duration: 6.02556953s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:17:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 6.025598145s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.24208594s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:17:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.242111621s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X........................................X..................] duration: 3.53377036s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:17:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533794806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X........................................X.................] duration: 3.537447692s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:16:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537474234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.........X...................] duration: 6.050066857s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:16:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 6.050100362s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.401849158s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:16:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.401882161s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X........................................X................] duration: 3.448078667s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:16:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448104496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.812892699s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:16:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.812918559s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.........X..................] duration: 5.975220091s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:15:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.975250368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X..X........................................X...............] duration: 3.653627948s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:15:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.653702169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.945434191s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:15:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.945507611s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.........X.................] duration: 5.949859632s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:15:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.949883749s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.368209704s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:14:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.368237656s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XX..X........................................X..............] duration: 3.46480063s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:14:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464841877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.........X................] duration: 5.786736569s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:14:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.786751527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.XX..X........................................X.............] duration: 3.661239142s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:13:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661267125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.75980561s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:13:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.759832381s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.........X...............] duration: 6.111061109s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 6.111095084s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.331636132s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:13:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.3317089s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..XX..X........................................X............] duration: 3.516620402s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:13:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516656391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.955054089s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:12:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.955091029s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X..XX..X........................................X...........] duration: 4.947103056s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:12:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.947128574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.........X..............] duration: 5.972961994s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:12:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.973028249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.........X.............] duration: 5.945340976s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:12:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.945370763s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.209100231s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:11:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.209130457s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X..XX..X........................................X..........] duration: 3.407233841s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:11:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407259238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.........X............] duration: 5.753559001s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:11:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.753587125s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.320017365s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:11:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.32004076s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.........X...........] duration: 7.545209072s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:10:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 7.545239971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X..XX..X........................................X.........] duration: 3.392481699s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:10:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.392543016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.327752367s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:10:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.327806279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.........X..........] duration: 5.684990644s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:09:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.685014751s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.238043823s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:09:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.238071455s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.267910385s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:09:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.267933109s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X..XX..X........................................X........] duration: 3.398600621s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:09:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398638133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.........X.........] duration: 29.398646819s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 29.398685884s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.804354039s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:08:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.804371934s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X..XX..X........................................X.......] duration: 3.355959826s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:08:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.355995193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X....X..XX..X........................................X......] duration: 5.001001897s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:07:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00102999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.........X........] duration: 11.516673949s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:07:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 11.516697183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XX....X..XX..X........................................X.....] duration: 5.001053054s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:07:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001127306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.626690738s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:07:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.626712599s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.XX....X..XX..X........................................X....] duration: 3.580467436s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:06:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580494318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.........X.......] duration: 5.835296525s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:06:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.83532998s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.346690156s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:06:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.346718981s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..XX....X..XX..X........................................X...] duration: 4.991677373s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:06:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.991709845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.........X......] duration: 5.98835729s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:05:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.988386165s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)