Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.362392947s trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:04:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 2.362428705s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....XX....X..XX..X........................................X.] duration: 5.001393584s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:04:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001428101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.........X....] duration: 5.683319372s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:04:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.683353777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....XX....X..XX..X........................................X] duration: 5.001232907s trace: https://jaeger.dev.storj.io/trace/44328df52239e1d6 FAIL (2025-12-16 11:03:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001256643s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 791.699567ms trace: https://jaeger.dev.storj.io/trace/55d2d7bcfe4d082d FAIL (2025-12-16 11:03:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44700->45.38.24.2:443: read: connection reset by peer Duration: 791.725657ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.........X...] duration: 5.934182201s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:03:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.934208419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.........X..] duration: 5.734136196s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:02:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.734162586s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......XX....X..XX..X........................................] duration: 4.109027769s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 11:02:45 UTC) orbiter warning: context deadline exceeded Duration: 4.10905436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......XX....X..XX..X.......................................] duration: 3.439946601s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 11:02:16 UTC) orbiter warning: context deadline exceeded Duration: 3.44003043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.........X.] duration: 5.92358127s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:01:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 5.923642909s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........XX....X..XX..X......................................] duration: 4.967603202s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 11:01:21 UTC) orbiter warning: context deadline exceeded Duration: 4.967632076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.........X] duration: 30.003305697s trace: https://jaeger.dev.storj.io/trace/5feb596856f9041d FAIL (2025-12-16 11:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ) context deadline exceeded ======LOGS====== Dec 16 11:00:48.151471190 (259h31m48.022554235s) [upload=227557]: Starting upload Dec 16 11:00:48.151547205 (259h31m48.02263025s) [upload=227557]: Done starting upload Dec 16 11:00:48.151595337 (259h31m48.022678372s) [upload=227557]: Uploading segments... Dec 16 11:00:48.151768046 (259h31m48.022851081s) [upload=227557]: Got next segment. Inline: false Dec 16 11:00:48.151823401 (259h31m48.022906456s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 11:00:48.151830034 (259h31m48.02291312s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 11:00:48.395917881 (259h31m48.267000916s) [seg_pos={0 0}, upload=227557]: Begin upload segment... Dec 16 11:00:48.395966845 (259h31m48.26704989s) [seg_pos={0 0}, upload=227557]: Done begin upload segment. Dec 16 11:00:48.395994266 (259h31m48.267077311s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396030665 (259h31m48.2671137s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.396583428 (259h31m48.267666473s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397071548 (259h31m48.268154593s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.397621595 (259h31m48.26870464s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398139011 (259h31m48.269222056s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.398788928 (259h31m48.269871973s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399236640 (259h31m48.270319685s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.399748495 (259h31m48.27083153s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.400883817 (259h31m48.271966852s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.402198210 (259h31m48.273281255s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.404841914 (259h31m48.27592496s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.407052964 (259h31m48.278136009s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.408057207 (259h31m48.279140242s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.409212596 (259h31m48.280295631s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:48.410538992 (259h31m48.281622027s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Uploading piece... Dec 16 11:00:50.098633515 (259h31m49.96971656s) [upload=227557]: Next returned nil segment Dec 16 11:00:50.098649064 (259h31m49.969732109s) [upload=227557]: Waiting for error group managing segments... Dec 16 11:00:50.379607329 (259h31m50.250690374s) [noise=true , address=145.79.227.32:4834 , piece_id=PXX3CQMULNRTHGKIPJHOLPEE5UM7BPYWFLVD66NO4MYFTMY3HXXQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.388098578 (259h31m50.259181613s) [noise=true , address=145.79.227.34:4907 , piece_id=4GWPKYJ3PO4J2TD2MV3BFUGKTFXDJJ67WEYXSJJ5FOHYEWR4ZPQQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.389717540 (259h31m50.260800575s) [noise=true , address=145.79.227.38:4977 , piece_id=5VVMO4RLZZH2ZFNL5UFPFYOKJX4TWUHWOHMJPGCCROAHLSVD57NQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.396333258 (259h31m50.267416303s) [noise=true , address=145.79.227.36:4518 , piece_id=LFL5PRNQMKL7466PNWGYV3EWEHBRRUVPMTVEI4GUYKANMQ2ZTERQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.397002503 (259h31m50.268085538s) [noise=true , address=145.79.227.49:4337 , piece_id=32UDPUJKMEL6QYYWN745DI2DPREH3YH7ML5TDRO6CFISK27XZOYA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.420033667 (259h31m50.291116703s) [noise=true , address=145.79.227.8:4331 , piece_id=XWESLSU7U6KDJS3CWJJO3YIPNJFBDI4F3WJYX47C5ZERI7UIMF3A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473139581 (259h31m50.344222626s) [noise=true , address=145.79.227.50:4375 , piece_id=WPE5J3I7VSXXQM7LN42CYCBB6KLORPO2IQZHWNGY4UHJH5NFZ2UQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.473671223 (259h31m50.344754268s) [noise=true , address=145.79.227.21:4475 , piece_id=23NBGBCNWDAC5DRTNLVUKWDLHYWVKSI7AHIXKZIEEGI3KRMHLJBQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.482689746 (259h31m50.353772781s) [noise=true , address=145.79.227.47:4266 , piece_id=QEGNLPRE2QRHCNDWOIXDDWWE5ACF4J3FSCTFX2T6RROBZPHFMFOA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.607196850 (259h31m50.478279895s) [noise=true , address=77.90.31.19:4842 , piece_id=OID2ZFGWUM3LM3WKO4KI3SKMI5STIDACYS65U2QIIEVDJMUKY5XQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.648498376 (259h31m50.519581421s) [noise=true , address=45.91.116.5:4300 , piece_id=PPAJU5DPTYLLWKXCTJNXV6IQHLO6HAMTBZMGQR77DT4JWEFWG2QQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.651814750 (259h31m50.522897785s) [noise=true , address=45.91.116.14:4190 , piece_id=6U26TI3ZOQ3ISONYRLF5QRKTOBNI64CKDEAGVFWE6JOWELO7AYNA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661083159 (259h31m50.532166204s) [noise=true , address=77.90.31.11:4554 , piece_id=A72GD5LPGSVUCOB4EVFT3AK7C4T6GEVDTLHRHQVDAVO63YTYZJUA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: <nil> Dec 16 11:00:50.661108497 (259h31m50.532191542s) [seg_pos={0 0}, upload=227557]: Segment reached optimal threshold of 13 pieces. Dec 16 11:00:50.661387879 (259h31m50.532470934s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661401986 (259h31m50.532485051s) [noise=true , address=77.90.31.10:4518 , piece_id=QBIKIIF7SYVAWREOK7CVJKRRQHPRJC5IUWUFHOEL7DQFXRHEQPXA, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661519340 (259h31m50.532602385s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661528617 (259h31m50.532611652s) [noise=true , address=145.79.227.30:4761 , piece_id=5E525CVUOYOTSOFYUJFISH32Z5UG4TDN2MYJU3WABUY5LCZCPPFQ, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661734549 (259h31m50.532817604s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 16 11:00:50.661743987 (259h31m50.532827032s) [noise=true , address=45.91.116.6:4975 , piece_id=P4CUXIODAIDASBOXG7KZGIYOY26PMEVOI5HA223DDZUP75DFZD4A, seg_pos={0 0}, upload=227557]: Long tail cancellation due to optimal threshold Dec 16 11:00:50.661765358 (259h31m50.532848403s) [seg_pos={0 0}, upload=227557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 11:00:50.661926084 (259h31m50.533009129s) [upload=227557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 11:00:50.661933378 (259h31m50.533016423s) [upload=227557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 11:00:50.950220799 (259h31m50.821303844s) [upload=227557]: Done uploading segments... Dec 16 11:00:50.950242249 (259h31m50.821325284s) [upload=227557]: Upload finished. err: <nil> Duration: 30.003340614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.917594476s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 11:00:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.917630364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........XX....X..XX..X.....................................] duration: 3.654345878s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 11:00:09 UTC) orbiter warning: context deadline exceeded Duration: 3.654380463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.736388103s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.736417199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........XX....X..XX..X....................................] duration: 3.570489387s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:59:42 UTC) orbiter warning: context deadline exceeded Duration: 3.570514756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.985107219s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.985146565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.159711376s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:58:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 6.159739328s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........XX....X..XX..X...................................] duration: 3.605133454s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:58:37 UTC) orbiter warning: context deadline exceeded Duration: 3.605159264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............XX....X..XX..X..................................] duration: 3.615838981s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:58:16 UTC) orbiter warning: context deadline exceeded Duration: 3.61587058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.692949719s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:58:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.692981809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............XX....X..XX..X.................................] duration: 3.508054629s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:57:47 UTC) orbiter warning: context deadline exceeded Duration: 3.50807656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.674685399s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.674714943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............XX....X..XX..X................................] duration: 4.311594931s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:56:49 UTC) orbiter warning: context deadline exceeded Duration: 4.311627193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.19725849s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 6.197306701s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............XX....X..XX..X...............................] duration: 3.497666839s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:56:10 UTC) orbiter warning: context deadline exceeded Duration: 3.49769419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.984021912s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.984056228s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................XX....X..XX..X..............................] duration: 3.55511999s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:55:33 UTC) orbiter warning: context deadline exceeded Duration: 3.555147442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.746116375s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:54:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 5.746143727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................XX....X..XX..X.............................] duration: 3.702583187s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:54:29 UTC) orbiter warning: context deadline exceeded Duration: 3.702606921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003696526s trace: https://jaeger.dev.storj.io/trace/7091af554d1ff831 FAIL (2025-12-16 10:54:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ) context deadline exceeded ======LOGS====== 6 10:53:34.643962816 (259h24m34.515045861s) [upload=227453]: Starting upload Dec 16 10:53:34.644059670 (259h24m34.515142725s) [upload=227453]: Done starting upload Dec 16 10:53:34.644168909 (259h24m34.515251954s) [upload=227453]: Uploading segments... Dec 16 10:53:34.644279469 (259h24m34.515362504s) [upload=227453]: Got next segment. Inline: false Dec 16 10:53:34.644321329 (259h24m34.515404374s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 10:53:34.644326719 (259h24m34.515409754s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 10:53:34.900237759 (259h24m34.771320824s) [seg_pos={0 0}, upload=227453]: Begin upload segment... Dec 16 10:53:34.900298295 (259h24m34.77138133s) [seg_pos={0 0}, upload=227453]: Done begin upload segment. Dec 16 10:53:34.900372345 (259h24m34.771455391s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900380330 (259h24m34.771463375s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.900994360 (259h24m34.772077395s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.901675637 (259h24m34.772758682s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902551806 (259h24m34.773634841s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.902561123 (259h24m34.773644158s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903014767 (259h24m34.774097802s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903219958 (259h24m34.774302993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903390393 (259h24m34.774473438s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903747082 (259h24m34.774830127s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.903820090 (259h24m34.774903126s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904229100 (259h24m34.775312135s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.904806929 (259h24m34.775889964s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905522412 (259h24m34.776605447s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.905965757 (259h24m34.777048792s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:34.907597574 (259h24m34.778680609s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Uploading piece... Dec 16 10:53:36.605538585 (259h24m36.47662163s) [upload=227453]: Next returned nil segment Dec 16 10:53:36.605555487 (259h24m36.476638542s) [upload=227453]: Waiting for error group managing segments... Dec 16 10:53:36.884933502 (259h24m36.756016537s) [noise=true , address=145.79.227.40:4051 , piece_id=N4DYUKZZY7TYDAB7W2ZULMHZ67PDK6DTPV7ID3P2ULOFRHGVL3MQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.895778213 (259h24m36.766861258s) [noise=true , address=145.79.227.22:4509 , piece_id=MWVAZWTG4YV7Z7BATC2T4PH3TX2JDYUGRGVTLXAG74UQ5MME3IXQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904033252 (259h24m36.775116287s) [noise=true , address=145.79.227.45:4229 , piece_id=MW6TPFMF2ZLOBXWJ3QNJSJ3IRDQE7YZZNEHOVQ7RBCDNB5CTVTEA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.904321491 (259h24m36.775404536s) [noise=true , address=145.79.227.34:4906 , piece_id=RNPP43RXBKP3GYYVZWW4HJVIHDU3K76M25KZED3SICZFGVVEN62A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918856168 (259h24m36.789939213s) [noise=true , address=145.79.227.41:4086 , piece_id=T7XQ7YATIOSESDOBNXGY6KIENMCFD6VAKSOHBUJUTM74ZJZ4VT5Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.918886315 (259h24m36.78996935s) [noise=true , address=145.79.227.12:4765 , piece_id=EHT6JJV2D2FCFGZHAIREB6HLVK3MX2IUX4WVVNY3QFDPTNIHHKOA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919228517 (259h24m36.790311552s) [noise=true , address=145.79.227.29:4942 , piece_id=RZYQBLW77INFHAVOQ62OU3U3VOP66OH6UQ2BXVD5QIP4DG72JHBQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.919939971 (259h24m36.791023006s) [noise=true , address=145.79.227.13:4804 , piece_id=TTIPHHQAZOJRTSWVKAPVCE2BFRAEBW4KV6BBCRRU7YMLSLMUOHHA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.977975871 (259h24m36.849058906s) [noise=true , address=145.79.227.23:4545 , piece_id=MULF7PQRLQTGEWOAP46MXOQPF6TK2EUBOSIQGUL4TODC527QBYDQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996380948 (259h24m36.867463993s) [noise=true , address=145.79.227.35:4482 , piece_id=HK7S4YRQCXW6I3C2AFPJUWUGNQKAVZJV2N57GDK476EMW5OMPU2Q, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:36.996808502 (259h24m36.867891537s) [noise=true , address=145.79.227.50:4373 , piece_id=BP6D2R4L7RUWFBDAK4VD5AWRJGFJH7KDHYFAJGDJVO2DK27GYESQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.164425004 (259h24m37.035508039s) [noise=true , address=45.91.116.6:4975 , piece_id=HGDNOX5H4AL2MIFJ6LTTYGRJCQ6TFM5OB7EO6MNGCAGM7JENGXGQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166717098 (259h24m37.037800133s) [noise=true , address=45.91.116.8:4046 , piece_id=IJSL5RXH7O5QCY6HM2OBEWCWEQSTE7MJRM7NDRLEX2JEDNRWVP4A, seg_pos={0 0}, upload=227453]: Done uploading piece. err: <nil> Dec 16 10:53:37.166735554 (259h24m37.037818589s) [seg_pos={0 0}, upload=227453]: Segment reached optimal threshold of 13 pieces. Dec 16 10:53:37.166921097 (259h24m37.038004142s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.166927860 (259h24m37.038010895s) [noise=true , address=77.90.31.21:4880 , piece_id=G6Q73MFMXO3W3MGVTA3U5XJZ3POLOMV2KVONL25FTL35BAVMKMNA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167150714 (259h24m37.038233759s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 16 10:53:37.167156856 (259h24m37.038239891s) [noise=true , address=77.90.31.11:4556 , piece_id=74TYTJYCQ25CWKUFTNCUS6JFCCP4JJYXUKCT6O3IHFMJ4GJL6VVQ, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167266254 (259h24m37.038349289s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37546->45.91.116.13:4156: use of closed network connection Dec 16 10:53:37.167271524 (259h24m37.038354569s) [noise=true , address=45.91.116.13:4156 , piece_id=KSTIQLOC2V2CNJZASMN34UAUSH2SID7MI4POEPGYRUCONVGFWOMA, seg_pos={0 0}, upload=227453]: Long tail cancellation due to optimal threshold Dec 16 10:53:37.167287674 (259h24m37.038370709s) [seg_pos={0 0}, upload=227453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 10:53:37.167353059 (259h24m37.038436104s) [upload=227453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 10:53:37.167357918 (259h24m37.038440953s) [upload=227453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 10:53:37.468506709 (259h24m37.339589754s) [upload=227453]: Done uploading segments... Dec 16 10:53:37.468627610 (259h24m37.339710645s) [upload=227453]: Upload finished. err: <nil> Duration: 30.003717336s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................XX....X..XX..X............................] duration: 3.623363145s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:54:01 UTC) orbiter warning: context deadline exceeded Duration: 3.623388905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...................XX....X..XX..X...........................] duration: 3.561125715s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:53:32 UTC) orbiter warning: context deadline exceeded Duration: 3.561150072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................XX....X..XX..X..........................] duration: 3.868081853s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:53:04 UTC) orbiter warning: context deadline exceeded Duration: 3.868115026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....................XX....X..XX..X.........................] duration: 3.635564098s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:52:25 UTC) orbiter warning: context deadline exceeded Duration: 3.635587463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................XX....X..XX..X........................] duration: 3.528535269s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:51:56 UTC) orbiter warning: context deadline exceeded Duration: 3.528585885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......................XX....X..XX..X.......................] duration: 3.579854059s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:51:06 UTC) orbiter warning: context deadline exceeded Duration: 3.579882793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........................XX....X..XX..X......................] duration: 3.544475773s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:49:55 UTC) orbiter warning: context deadline exceeded Duration: 3.54450132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........................XX....X..XX..X.....................] duration: 3.648830459s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:49:01 UTC) orbiter warning: context deadline exceeded Duration: 3.648856689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........................XX....X..XX..X....................] duration: 3.473127532s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:48:13 UTC) orbiter warning: context deadline exceeded Duration: 3.473157779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........................XX....X..XX..X...................] duration: 3.59583643s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:47:03 UTC) orbiter warning: context deadline exceeded Duration: 3.59586794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............................XX....X..XX..X..................] duration: 3.428147822s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:46:44 UTC) orbiter warning: context deadline exceeded Duration: 3.42817291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............................XX....X..XX..X.................] duration: 3.584368924s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:45:42 UTC) orbiter warning: context deadline exceeded Duration: 3.584396146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............................XX....X..XX..X................] duration: 3.460607179s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:44:38 UTC) orbiter warning: context deadline exceeded Duration: 3.460637829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............................XX....X..XX..X...............] duration: 3.585790923s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:43:24 UTC) orbiter warning: context deadline exceeded Duration: 3.58582145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................................XX....X..XX..X..............] duration: 3.757765132s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:42:17 UTC) orbiter warning: context deadline exceeded Duration: 3.757799378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................................XX....X..XX..X.............] duration: 3.564704663s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:41:26 UTC) orbiter warning: context deadline exceeded Duration: 3.564730823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..................................XX....X..XX..X............] duration: 3.576461196s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:40:14 UTC) orbiter warning: context deadline exceeded Duration: 3.576492936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...................................XX....X..XX..X...........] duration: 3.579395951s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:39:51 UTC) orbiter warning: context deadline exceeded Duration: 3.579420118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................................XX....X..XX..X..........] duration: 3.584623458s trace: https://jaeger.dev.storj.io/trace/3ee3e69a6af76e4 FAIL (2025-12-16 10:39:17 UTC) orbiter warning: context deadline exceeded Duration: 3.584645783s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)