Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.431882618s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:11:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.43190927s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..XX.X.X....XX......X..X....X................XX....X........] duration: 4.067422198s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:11:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.067445243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.779413036s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:11:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.779440729s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.794105471s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:10:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.79413078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...XX.X.X....XX......X..X....X................XX....X.......] duration: 3.842627799s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:10:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.842699666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.292098694s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:10:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.292135945s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.100983821s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.101042633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.679720912s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:09:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.679776078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...XX.X.X....XX......X..X....X................XX....X......] duration: 3.588594354s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:09:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588631214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.015144821s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:09:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.015169267s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.95638886s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:09:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.956460507s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.862271781s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:08:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.862311486s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X...XX.X.X....XX......X..X....X................XX....X.....] duration: 3.471906658s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:08:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471940111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.386056449s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:08:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.38610984s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 4.158877187s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.158890602s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.896404115s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:08:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.896434664s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X...XX.X.X....XX......X..X....X................XX....X....] duration: 5.001326848s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:07:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001350363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.259802497s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:07:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.259828457s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.164618293s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:07:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.164640796s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X..X...XX.X.X....XX......X..X....X................XX....X...] duration: 3.518321783s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:06:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518364123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 4.180519195s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:06:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.180565042s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X..X...XX.X.X....XX......X..X....X................XX....X..] duration: 3.974173052s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:06:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.974199994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.961260818s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:06:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.961283192s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.754917289s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:06:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.754937838s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.833601923s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:05:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.833631599s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.917346082s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:05:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.917372302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.010397119s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:05:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.010427877s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X..X...XX.X.X....XX......X..X....X................XX....X.] duration: 3.491330417s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:05:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491357659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.305244744s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:05:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.305276083s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X..X..X...XX.X.X....XX......X..X....X................XX....X] duration: 5.001972429s trace: https://jaeger.dev.storj.io/trace/2a5ae08650c95641 FAIL (2025-12-16 17:04:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001999861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.332610308s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:04:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.332632531s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............................X............................] duration: 5.890498309s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:04:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.890564946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................X...........................] duration: 5.742229489s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:03:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.742253614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX..X..X...XX.X.X....XX......X..X....X................XX....] duration: 3.633349446s trace: https://jaeger.dev.storj.io/trace/a6e3db29f370ed FAIL (2025-12-16 17:03:14 UTC) orbiter warning: context deadline exceeded Duration: 3.633375395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.278424925s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:03:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.278449803s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXX..X..X...XX.X.X....XX......X..X....X................XX...] duration: 5.00062842s trace: https://jaeger.dev.storj.io/trace/a6e3db29f370ed FAIL (2025-12-16 17:02:58 UTC) orbiter warning: context deadline exceeded Duration: 5.00064402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 3.438897254s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:02:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438922952s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..............................X..........................] duration: 6.014835742s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:02:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.014867402s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXXX..X..X...XX.X.X....XX......X..X....X................XX..] duration: 4.901013485s trace: https://jaeger.dev.storj.io/trace/a6e3db29f370ed FAIL (2025-12-16 17:02:14 UTC) orbiter warning: context deadline exceeded Duration: 4.901040316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..............................X.........................] duration: 5.781804038s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:02:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.781837191s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.366560393s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:01:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.366589398s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.XXXX..X..X...XX.X.X....XX......X..X....X................XX.] duration: 3.59554073s trace: https://jaeger.dev.storj.io/trace/a6e3db29f370ed FAIL (2025-12-16 17:01:24 UTC) orbiter warning: context deadline exceeded Duration: 3.595585665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..............................X........................] duration: 6.083779317s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:01:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.083812689s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 3.651052041s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:01:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.651080146s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.XXXX..X..X...XX.X.X....XX......X..X....X................XX] duration: 5.001786721s trace: https://jaeger.dev.storj.io/trace/a6e3db29f370ed FAIL (2025-12-16 17:00:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001812561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [X........................................X..................] duration: 2.757002995s trace: https://jaeger.dev.storj.io/trace/56c5b90ed930ab07 FAIL (2025-12-16 17:00:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.757043962s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..............................X.......................] duration: 5.682048672s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 17:00:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 5.682111131s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XX.XXXX..X..X...XX.X.X....XX......X..X....X................X] duration: 5.001862315s trace: https://jaeger.dev.storj.io/trace/44d330fefcf2838f FAIL (2025-12-16 16:59:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001912731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXX.XXXX..X..X...XX.X.X....XX......X..X....X................] duration: 3.486361102s trace: https://jaeger.dev.storj.io/trace/4516f25f16f6422b FAIL (2025-12-16 16:59:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486376991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..............................X......................] duration: 6.357481432s trace: https://jaeger.dev.storj.io/trace/26cc241f6bc0f273 FAIL (2025-12-16 16:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 16:42:35.170421692 (265h13m35.041504727s) [upload=232502]: Starting upload Dec 16 16:42:35.170534056 (265h13m35.041617101s) [upload=232502]: Done starting upload Dec 16 16:42:35.170664244 (265h13m35.041747279s) [upload=232502]: Uploading segments... Dec 16 16:42:35.170824980 (265h13m35.041908025s) [upload=232502]: Got next segment. Inline: false Dec 16 16:42:35.170879483 (265h13m35.041962528s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 16:42:35.170885806 (265h13m35.041968851s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 16:42:35.460872398 (265h13m35.331955433s) [seg_pos={0 0}, upload=232502]: Begin upload segment... Dec 16 16:42:35.460924156 (265h13m35.332007191s) [seg_pos={0 0}, upload=232502]: Done begin upload segment. Dec 16 16:42:35.460947701 (265h13m35.332030747s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461002586 (265h13m35.332085621s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.461367090 (265h13m35.332450125s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462268306 (265h13m35.333351341s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.462793717 (265h13m35.333876762s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463205491 (265h13m35.334288526s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463508358 (265h13m35.334591393s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.463801135 (265h13m35.33488418s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464705387 (265h13m35.335788422s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.464877795 (265h13m35.33596083s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.465832763 (265h13m35.336915808s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.466117286 (265h13m35.337200331s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.467520818 (265h13m35.338603853s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.468972753 (265h13m35.340055798s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.470463851 (265h13m35.341546897s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:35.471609754 (265h13m35.342692799s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Uploading piece... Dec 16 16:42:37.167397710 (265h13m37.038480745s) [upload=232502]: Next returned nil segment Dec 16 16:42:37.167411466 (265h13m37.038494501s) [upload=232502]: Waiting for error group managing segments... Dec 16 16:42:37.453956765 (265h13m37.3250398s) [noise=true , address=145.79.227.9:4365 , piece_id=CBOEDJ3M57NPUAQRD3J6YLLZ6HYA6RNRQUULFFE7CKJJWZNIK6AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.455150628 (265h13m37.326233663s) [noise=true , address=145.79.227.42:4123 , piece_id=B3QYM3OUDZ6SH52BERTSWBXGUCHOKQ2N2PMP5LDHJO2MBOFPBFNA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.460274556 (265h13m37.331357591s) [noise=true , address=145.79.227.35:4482 , piece_id=ST6LTQAGTVJZTQKXFOCWVPJXCXK4ZBLF3QHRUL47AC4OVM66OZPQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.463082462 (265h13m37.334165497s) [noise=true , address=145.79.227.38:4978 , piece_id=MX2QNYTFVVX76G4OMY4AEFJMFOUPHGWU2MEYYYHIRB2B2GTLXM3A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.464312264 (265h13m37.335395309s) [noise=true , address=145.79.227.8:4330 , piece_id=YE63RBCIWMDPUKFFYX7LDBW3GHBZHBEAMG2MI4I2CQXGJHYFWWQQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.466513055 (265h13m37.3375961s) [noise=true , address=145.79.227.36:4518 , piece_id=W4Z2M6R4UK5FEYJEUZEB5R2SBKNPVB3SBYJFUPGWUW72LAAHNU7A, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.477345361 (265h13m37.348428396s) [noise=true , address=145.79.227.48:4302 , piece_id=YVUEN33FADUX2SL6C2OAAFTHYC2Q7E5L3TELZHG43WAU27Z7VULQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.479877754 (265h13m37.350960789s) [noise=true , address=145.79.227.5:4222 , piece_id=SUJPJ4JVGR2MSIL6CR2CTTUW6XJHTMFTXTYIZTMVH7BB4FOSQATQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.483004758 (265h13m37.354087793s) [noise=true , address=145.79.227.27:4689 , piece_id=EFNAUZO6I5YSUPLO6ENMGBZQTKFNFC6MNTHYMXMBZNUOSBTYYZOA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485395790 (265h13m37.356478825s) [noise=true , address=145.79.227.37:4566 , piece_id=7VTOCUQUCHJINWCJL32T4NWTUVTCS2YKLYINVDASTBZPFNAJ3LHQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.485907425 (265h13m37.35699046s) [noise=true , address=145.79.227.10:4401 , piece_id=XOTZ4L6JCEIF4G6VB6LHXLTPRSGBNWF7PPNVQ27TSOHCIG352OWA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.720164128 (265h13m37.591247163s) [noise=true , address=45.91.116.6:4973 , piece_id=GXBZFMN7JC4GOT25CJWMBDGQHWX4HEWQDOTO2AC27JYMMYRHZ5AA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741631836 (265h13m37.612714881s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BNZEPNTMTQH2E7K4XPWDQD4QMJ2JP34IN2VG35WBFLM4AF24JA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: <nil> Dec 16 16:42:37.741657614 (265h13m37.61274066s) [seg_pos={0 0}, upload=232502]: Segment reached optimal threshold of 13 pieces. Dec 16 16:42:37.741971532 (265h13m37.613054577s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.741984858 (265h13m37.613067933s) [noise=true , address=45.91.116.10:4083 , piece_id=GM6MYBXR6TQCIQY2LQV7AHOBV4RLHKXDPM7N55OI22JVEH233IYA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742460974 (265h13m37.613544019s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742474300 (265h13m37.613557375s) [noise=true , address=45.91.116.13:4156 , piece_id=ZP2X2TIHXMT6MVDQRG2BEZKJKG3S7NRQFHJOQS46JHTXIJIMBFXQ, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742693497 (265h13m37.613776562s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 16 16:42:37.742703927 (265h13m37.613786972s) [noise=true , address=145.79.227.33:4870 , piece_id=MLPC576XSLRMVVDWMXOHGYHXYIED6KR76DZLYLKR42C4AOBORWIA, seg_pos={0 0}, upload=232502]: Long tail cancellation due to optimal threshold Dec 16 16:42:37.742730387 (265h13m37.613813432s) [seg_pos={0 0}, upload=232502]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 16:42:37.742872538 (265h13m37.613955593s) [upload=232502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 16:42:37.742881505 (265h13m37.61396455s) [upload=232502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 16:42:38.044853700 (265h13m37.915936735s) [upload=232502]: Done uploading segments... Dec 16 16:42:38.044873928 (265h13m37.915956973s) [upload=232502]: Upload finished. err: <nil> Duration: 6.357519125s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)