Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
3.590877076s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:42:43 UTC)
orbiter warning: context deadline exceeded
Duration: 3.590903167s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...........X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XX] duration:
30.002500293s trace:
https://jaeger.dev.storj.io/trace/3a75ed4065de85a8
FAIL (2025-12-26 08:42:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3U4VEU3U5LGNREBFQDS27AX66FAADSK7RYCI2FSQ4SCOMYYLNN4A) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: K3GGNL2ATRKFNVGJ63OXRCHHXZG2JO3JJU6RVDZ5M5D2Q3NN2LJQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: JYMIYZGEYORKZL7RXT7QINX5RXVBBIH735Q7JW22AKZFIOFDL7UA) context deadline exceeded
======LOGS======
Dec 26 08:41:39.754811332 (497h12m39.625894377s) [upload=436385]: Starting upload
Dec 26 08:41:39.754890433 (497h12m39.625973478s) [upload=436385]: Done starting upload
Dec 26 08:41:39.755100653 (497h12m39.626183698s) [upload=436385]: Uploading segments...
Dec 26 08:41:39.755164044 (497h12m39.626247099s) [upload=436385]: Got next segment. Inline: false
Dec 26 08:41:39.755202296 (497h12m39.626285331s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:41:39.755208348 (497h12m39.626291383s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:41:39.999183890 (497h12m39.870266935s) [seg_pos={0 0}, upload=436385]: Begin upload segment...
Dec 26 08:41:39.999243493 (497h12m39.870326529s) [seg_pos={0 0}, upload=436385]: Done begin upload segment.
Dec 26 08:41:39.999281927 (497h12m39.870364982s) [noise=true , address=145.79.227.5:4223 , piece_id=2KC4WMA745RUDV6NTP2NH74TOOXIXCCV72ICWK6H57JT7LI3IWSQ, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:39.999296704 (497h12m39.870379739s) [noise=true , address=145.79.227.19:4079 , piece_id=KVDN5AY5FJ4DKKAOSAMPTAKXA3BW57GV2DMXDVJ65KECUZJS2Z3Q, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:39.999957342 (497h12m39.871040377s) [noise=true , address=77.90.31.15:4697 , piece_id=U3JANLEUA77LV5NNEXMFE5UTDYQEZZRVNUL6ELNMDJMQ7B7XS5KA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.000106918 (497h12m39.871189953s) [noise=true , address=145.79.227.32:4834 , piece_id=OCZGVYEKHMJXE5CO663MPJLF7ZWUQUTZT24TMOC5YVXRFQQ2JJCA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.000513191 (497h12m39.871596236s) [noise=true , address=145.79.227.10:4403 , piece_id=TMEUYHOB2Y6LXOQZKDK6JA7IHZSI5MG63G4CHNRHE25G4VXLWG3A, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.000897593 (497h12m39.871980638s) [noise=true , address=45.91.116.14:4192 , piece_id=R75D4UOQPUDFYDSIWKQZNDKGJAEGT2RKVHWLDONYQSBGS7FKNLVA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.001258200 (497h12m39.872341245s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3QXNY3JN2QLSFJUUHE3C5UIOFYGFS2GTSP635IMK6USH7TVTUXA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.001607315 (497h12m39.87269035s) [noise=true , address=145.79.227.33:4869 , piece_id=GLMBU5HVI7FDDGD2M3L5ML56TGNSENZ3DIWTKCOAFU5GCF4TNFRA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.002034117 (497h12m39.873117162s) [noise=true , address=45.91.116.6:4975 , piece_id=3U4VEU3U5LGNREBFQDS27AX66FAADSK7RYCI2FSQ4SCOMYYLNN4A, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.002491568 (497h12m39.873574603s) [noise=true , address=145.79.227.38:4979 , piece_id=RSTUD5XUH7BR4MLNZEF6RCTEQHX4JR45EOROZOENAZDZDJN63U4A, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.002921537 (497h12m39.874004573s) [noise=true , address=45.91.116.11:4119 , piece_id=EB3YNXKLDOHSYBTME3RXS5DO6SVIXRJNVZD2NSC5RIL35K6WUBMQ, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.003321178 (497h12m39.874404233s) [noise=true , address=77.90.31.11:4553 , piece_id=PRTIXAKHZLRZDLVOTSQEBXGAR2KKE52EJB7VKMRSMVXB5ZJYMMKA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.003766276 (497h12m39.874849311s) [noise=true , address=45.91.116.12:4225 , piece_id=K3GGNL2ATRKFNVGJ63OXRCHHXZG2JO3JJU6RVDZ5M5D2Q3NN2LJQ, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.004202786 (497h12m39.875285831s) [noise=true , address=145.79.227.43:4158 , piece_id=AT24CKYX343R63HDR3FYKEJG7A6IHVWXE3B46WD3R6NZ6MCAHSHA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.004681298 (497h12m39.875764343s) [noise=true , address=77.90.31.10:4519 , piece_id=JYMIYZGEYORKZL7RXT7QINX5RXVBBIH735Q7JW22AKZFIOFDL7UA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.005185338 (497h12m39.876268373s) [noise=true , address=145.79.227.13:4802 , piece_id=EOQH6Z3DNHYSJMVHHBWVHFBSF4JQKEEQVVEPPF6TYXBDV7OF7BFA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:41.225124735 (497h12m41.09620777s) [noise=true , address=77.90.31.15:4697 , piece_id=U3JANLEUA77LV5NNEXMFE5UTDYQEZZRVNUL6ELNMDJMQ7B7XS5KA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000
Dec 26 08:41:41.722443478 (497h12m41.593526513s) [upload=436385]: Next returned nil segment
Dec 26 08:41:41.722454699 (497h12m41.593537734s) [upload=436385]: Waiting for error group managing segments...
Dec 26 08:41:42.003644463 (497h12m41.874727498s) [noise=true , address=145.79.227.13:4802 , piece_id=EOQH6Z3DNHYSJMVHHBWVHFBSF4JQKEEQVVEPPF6TYXBDV7OF7BFA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.018308696 (497h12m41.889391741s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3QXNY3JN2QLSFJUUHE3C5UIOFYGFS2GTSP635IMK6USH7TVTUXA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.019347815 (497h12m41.89043086s) [noise=true , address=145.79.227.38:4979 , piece_id=RSTUD5XUH7BR4MLNZEF6RCTEQHX4JR45EOROZOENAZDZDJN63U4A, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.027768640 (497h12m41.898851675s) [noise=true , address=145.79.227.5:4223 , piece_id=2KC4WMA745RUDV6NTP2NH74TOOXIXCCV72ICWK6H57JT7LI3IWSQ, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.034410327 (497h12m41.905493362s) [noise=true , address=145.79.227.19:4079 , piece_id=KVDN5AY5FJ4DKKAOSAMPTAKXA3BW57GV2DMXDVJ65KECUZJS2Z3Q, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.034492924 (497h12m41.905575959s) [noise=true , address=145.79.227.10:4403 , piece_id=TMEUYHOB2Y6LXOQZKDK6JA7IHZSI5MG63G4CHNRHE25G4VXLWG3A, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.042390482 (497h12m41.913473527s) [noise=true , address=145.79.227.32:4834 , piece_id=OCZGVYEKHMJXE5CO663MPJLF7ZWUQUTZT24TMOC5YVXRFQQ2JJCA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.044373167 (497h12m41.915456213s) [noise=true , address=145.79.227.43:4158 , piece_id=AT24CKYX343R63HDR3FYKEJG7A6IHVWXE3B46WD3R6NZ6MCAHSHA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.361607968 (497h12m42.232691013s) [noise=true , address=77.90.31.10:4519 , piece_id=JYMIYZGEYORKZL7RXT7QINX5RXVBBIH735Q7JW22AKZFIOFDL7UA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.371305083 (497h12m42.242388118s) [noise=true , address=45.91.116.6:4975 , piece_id=3U4VEU3U5LGNREBFQDS27AX66FAADSK7RYCI2FSQ4SCOMYYLNN4A, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.452064644 (497h12m42.323147679s) [noise=true , address=45.91.116.12:4225 , piece_id=K3GGNL2ATRKFNVGJ63OXRCHHXZG2JO3JJU6RVDZ5M5D2Q3NN2LJQ, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.599995399 (497h12m42.471078434s) [noise=true , address=45.91.116.14:4192 , piece_id=R75D4UOQPUDFYDSIWKQZNDKGJAEGT2RKVHWLDONYQSBGS7FKNLVA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:43.151072065 (497h12m43.0221551s) [noise=true , address=45.91.116.11:4119 , piece_id=EB3YNXKLDOHSYBTME3RXS5DO6SVIXRJNVZD2NSC5RIL35K6WUBMQ, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:43.151095160 (497h12m43.022178205s) [seg_pos={0 0}, upload=436385]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:41:43.151328473 (497h12m43.022411508s) [noise=true , address=145.79.227.33:4869 , piece_id=GLMBU5HVI7FDDGD2M3L5ML56TGNSENZ3DIWTKCOAFU5GCF4TNFRA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:41:43.151338433 (497h12m43.022421478s) [noise=true , address=145.79.227.33:4869 , piece_id=GLMBU5HVI7FDDGD2M3L5ML56TGNSENZ3DIWTKCOAFU5GCF4TNFRA, seg_pos={0 0}, upload=436385]: Long tail cancellation due to optimal threshold
Dec 26 08:41:43.151426500 (497h12m43.022509545s) [noise=true , address=77.90.31.11:4553 , piece_id=PRTIXAKHZLRZDLVOTSQEBXGAR2KKE52EJB7VKMRSMVXB5ZJYMMKA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:53494->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 08:41:43.151447791 (497h12m43.022530846s) [noise=true , address=77.90.31.11:4553 , piece_id=PRTIXAKHZLRZDLVOTSQEBXGAR2KKE52EJB7VKMRSMVXB5ZJYMMKA, seg_pos={0 0}, upload=436385]: Long tail cancellation due to optimal threshold
Dec 26 08:41:43.151547070 (497h12m43.022630115s) [seg_pos={0 0}, upload=436385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:41:43.151650798 (497h12m43.022733843s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:41:43.151656569 (497h12m43.022739604s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:41:43.440039961 (497h12m43.311123006s) [upload=436385]: Done uploading segments...
Dec 26 08:41:43.440077573 (497h12m43.311160608s) [upload=436385]: Upload finished. err: <nil>
Duration: 30.002526343s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
3.453528863s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:41:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.453589508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.244095448s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:41:51 UTC)
orbiter warning: context deadline exceeded
Duration: 6.244121628s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
32.776184571s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:41:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 32.776197916s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
52.026410745s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:41:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 52.026492672s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
3.405256103s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:40:59 UTC)
orbiter warning: context deadline exceeded
Duration: 3.405283584s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [............X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.X] duration:
30.040387447s trace:
https://jaeger.dev.storj.io/trace/2968e132c033ad32
FAIL (2025-12-26 08:40:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:40:29.189609540 (497h11m29.060692585s) [upload=436369]: Starting upload
Dec 26 08:40:29.189699512 (497h11m29.060782547s) [upload=436369]: Done starting upload
Dec 26 08:40:29.189864135 (497h11m29.06094717s) [upload=436369]: Uploading segments...
Dec 26 08:40:29.189971811 (497h11m29.061054856s) [upload=436369]: Got next segment. Inline: false
Dec 26 08:40:29.190023549 (497h11m29.061106584s) [upload=436369]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:40:29.190030572 (497h11m29.061113617s) [upload=436369]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:40:29.432637881 (497h11m29.303720926s) [seg_pos={0 0}, upload=436369]: Begin upload segment...
Dec 26 08:40:29.432704588 (497h11m29.303787623s) [seg_pos={0 0}, upload=436369]: Done begin upload segment.
Dec 26 08:40:29.432765964 (497h11m29.303849009s) [noise=true , address=45.91.116.11:4120 , piece_id=XN3RB6BTHT7PNZS3NIVSY6X224FHS3I2ZVZAJXWDLLQW6DAFFYQA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.432770624 (497h11m29.303853669s) [noise=true , address=145.79.227.5:4223 , piece_id=YJW3XDMQBFWIWY4EVWYBE4XCVA7EMC6CO4IAB3AST3YGEKBDYOUQ, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.433227373 (497h11m29.304310418s) [noise=true , address=145.79.227.47:4266 , piece_id=EXBKPFQBSVSEPZSHKBBQMIL625APZVDH3GF5JHQQ4N76R5KX6TAQ, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.433520281 (497h11m29.304603316s) [noise=true , address=45.91.116.14:4192 , piece_id=VWOML6LANB5NTVQ3SDR6BDIWWSK325GLPKZT43BT73ID3RTS4RVQ, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.433831193 (497h11m29.304914228s) [noise=true , address=145.79.227.46:4602 , piece_id=2O672LNRAOSSEXUHV44SZYYEXU7RVPJ5KWRPZFDFP2XYI2Z3QHHA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.434298723 (497h11m29.305381768s) [noise=true , address=145.79.227.29:4941 , piece_id=6TNPVGJEYQFHKY6TQJFQM2L3ROQX5T3LNOKIQLKWZ3KPMTEUELGA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.434670341 (497h11m29.305753376s) [noise=true , address=145.79.227.35:4481 , piece_id=EI4VCLA6CZLVWKJ7XK4E3QKF3RC2WAKVAVI6DYDO5PK6FPFW332A, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.434999288 (497h11m29.306082323s) [noise=true , address=45.91.116.5:4300 , piece_id=XCTXVU2RVZIHWKTRL56BAC5FOFF2N4JPLMTU5QFU2S4JHK23PFPQ, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.435318205 (497h11m29.30640124s) [noise=true , address=145.79.227.32:4834 , piece_id=MSZBJYDOBGESFJTCZ2HLTDUJQSS247ZUDOOWXSICUNBQBI75WNXQ, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.435758092 (497h11m29.306841137s) [noise=true , address=145.79.227.49:4339 , piece_id=NIV7IBJTGRDALNR3GN7JQNPB6SHIZSH3QSDMJMIIE3T4P2UX4WNA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.436179724 (497h11m29.307262759s) [noise=true , address=145.79.227.13:4803 , piece_id=77GQXPQZCUQEYQ6O4IF4CEWDD2OTVGP37IB2WP7R6KKD4X2ZPGTA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.436575338 (497h11m29.307658373s) [noise=true , address=45.91.116.7:4011 , piece_id=TNTXBTLXMNMJWIRZWOAWRIEQAB62Z23FPM5SSHASY74TJDYOGPUA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.436650591 (497h11m29.307733637s) [noise=true , address=45.91.116.12:4228 , piece_id=GRMTV6QGJEHZB37GLKSY5KWVXX66JT3R6I2HT73OFT3Z333ZTPIA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.437023512 (497h11m29.308106547s) [noise=true , address=145.79.227.42:4121 , piece_id=4X3PL5CEGM47WGMCVMRG6SM2K27PT5FY5HRALBF3LR4HKPEQ7UVA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.437474260 (497h11m29.308557295s) [noise=true , address=145.79.227.41:4085 , piece_id=D3UVKMCXU3A5SPWHACZCIM3AOIQ4NDJNFALDQGIC4VCHD7VP5JNQ, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:29.437527631 (497h11m29.308610666s) [noise=true , address=145.79.227.26:4655 , piece_id=7TWWYF3XCK77Y7VNYZSD5HIGFORJ47LACBDOZ5EMSNEPXT2YM2RA, seg_pos={0 0}, upload=436369]: Uploading piece...
Dec 26 08:40:31.145599559 (497h11m31.016682594s) [upload=436369]: Next returned nil segment
Dec 26 08:40:31.145613856 (497h11m31.016696901s) [upload=436369]: Waiting for error group managing segments...
Dec 26 08:40:31.435186412 (497h11m31.306269457s) [noise=true , address=145.79.227.29:4941 , piece_id=6TNPVGJEYQFHKY6TQJFQM2L3ROQX5T3LNOKIQLKWZ3KPMTEUELGA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.436755871 (497h11m31.307838916s) [noise=true , address=145.79.227.49:4339 , piece_id=NIV7IBJTGRDALNR3GN7JQNPB6SHIZSH3QSDMJMIIE3T4P2UX4WNA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.436959438 (497h11m31.308042473s) [noise=true , address=145.79.227.41:4085 , piece_id=D3UVKMCXU3A5SPWHACZCIM3AOIQ4NDJNFALDQGIC4VCHD7VP5JNQ, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.438979475 (497h11m31.31006251s) [noise=true , address=145.79.227.13:4803 , piece_id=77GQXPQZCUQEYQ6O4IF4CEWDD2OTVGP37IB2WP7R6KKD4X2ZPGTA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.440309607 (497h11m31.311392643s) [noise=true , address=145.79.227.32:4834 , piece_id=MSZBJYDOBGESFJTCZ2HLTDUJQSS247ZUDOOWXSICUNBQBI75WNXQ, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.440791515 (497h11m31.31187455s) [noise=true , address=145.79.227.47:4266 , piece_id=EXBKPFQBSVSEPZSHKBBQMIL625APZVDH3GF5JHQQ4N76R5KX6TAQ, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.443083289 (497h11m31.314166324s) [noise=true , address=145.79.227.5:4223 , piece_id=YJW3XDMQBFWIWY4EVWYBE4XCVA7EMC6CO4IAB3AST3YGEKBDYOUQ, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.443289733 (497h11m31.314372778s) [noise=true , address=145.79.227.42:4121 , piece_id=4X3PL5CEGM47WGMCVMRG6SM2K27PT5FY5HRALBF3LR4HKPEQ7UVA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.444096609 (497h11m31.315179654s) [noise=true , address=145.79.227.46:4602 , piece_id=2O672LNRAOSSEXUHV44SZYYEXU7RVPJ5KWRPZFDFP2XYI2Z3QHHA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.447422753 (497h11m31.318505788s) [noise=true , address=145.79.227.35:4481 , piece_id=EI4VCLA6CZLVWKJ7XK4E3QKF3RC2WAKVAVI6DYDO5PK6FPFW332A, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.459901013 (497h11m31.330984058s) [noise=true , address=145.79.227.26:4655 , piece_id=7TWWYF3XCK77Y7VNYZSD5HIGFORJ47LACBDOZ5EMSNEPXT2YM2RA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.688696750 (497h11m31.559779785s) [noise=true , address=45.91.116.7:4011 , piece_id=TNTXBTLXMNMJWIRZWOAWRIEQAB62Z23FPM5SSHASY74TJDYOGPUA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.702722908 (497h11m31.573805964s) [noise=true , address=45.91.116.12:4228 , piece_id=GRMTV6QGJEHZB37GLKSY5KWVXX66JT3R6I2HT73OFT3Z333ZTPIA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: <nil>
Dec 26 08:40:31.702749469 (497h11m31.573832514s) [seg_pos={0 0}, upload=436369]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:40:31.702964078 (497h11m31.574047123s) [noise=true , address=45.91.116.11:4120 , piece_id=XN3RB6BTHT7PNZS3NIVSY6X224FHS3I2ZVZAJXWDLLQW6DAFFYQA, seg_pos={0 0}, upload=436369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 26 08:40:31.702976982 (497h11m31.574060027s) [noise=true , address=45.91.116.11:4120 , piece_id=XN3RB6BTHT7PNZS3NIVSY6X224FHS3I2ZVZAJXWDLLQW6DAFFYQA, seg_pos={0 0}, upload=436369]: Long tail cancellation due to optimal threshold
Dec 26 08:40:31.703131747 (497h11m31.574214782s) [noise=true , address=45.91.116.14:4192 , piece_id=VWOML6LANB5NTVQ3SDR6BDIWWSK325GLPKZT43BT73ID3RTS4RVQ, seg_pos={0 0}, upload=436369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 26 08:40:31.703148509 (497h11m31.574231544s) [noise=true , address=45.91.116.14:4192 , piece_id=VWOML6LANB5NTVQ3SDR6BDIWWSK325GLPKZT43BT73ID3RTS4RVQ, seg_pos={0 0}, upload=436369]: Long tail cancellation due to optimal threshold
Dec 26 08:40:31.703193144 (497h11m31.574276189s) [noise=true , address=45.91.116.5:4300 , piece_id=XCTXVU2RVZIHWKTRL56BAC5FOFF2N4JPLMTU5QFU2S4JHK23PFPQ, seg_pos={0 0}, upload=436369]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:40:31.703202953 (497h11m31.574285988s) [noise=true , address=45.91.116.5:4300 , piece_id=XCTXVU2RVZIHWKTRL56BAC5FOFF2N4JPLMTU5QFU2S4JHK23PFPQ, seg_pos={0 0}, upload=436369]: Long tail cancellation due to optimal threshold
Dec 26 08:40:31.703249551 (497h11m31.574332596s) [seg_pos={0 0}, upload=436369]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:40:31.703333011 (497h11m31.574416056s) [upload=436369]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:40:31.703339252 (497h11m31.574422297s) [upload=436369]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:40:31.993478468 (497h11m31.864561513s) [upload=436369]: Done uploading segments...
Dec 26 08:40:31.993507553 (497h11m31.864590588s) [upload=436369]: Upload finished. err: <nil>
Duration: 30.040412645s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.24559685s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:40:56 UTC)
orbiter warning: context deadline exceeded
Duration: 6.245636414s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
4.32481331s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:40:16 UTC)
orbiter warning: context deadline exceeded
Duration: 4.324853586s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.152216736s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:40:15 UTC)
orbiter warning: context deadline exceeded
Duration: 6.152295475s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.............X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.] duration:
5.946209935s trace:
https://jaeger.dev.storj.io/trace/2ebe8a728d6ae142
FAIL (2025-12-26 08:39:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: DATLF3Z2NGF3AHXNJM6PA3VAG4PWTIOM4ZXVHCWIVTSEGAJDNOPA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: F2JUFZV3C43BUJTI2MOXC5GTPAQEEQKTNHGBXSLPBYKZBKIMBUVQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: HF52OKJB4VMTMWSFPGSKY2VNROKY3KQAWX7F5SLNUTW4RN4FQQKQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: R5H4WYA6OGWUWNTPZQMI7XVVRATVMKCKLEI2L2JVSSK5F773RT6A) context deadline exceeded
======LOGS======
Dec 26 08:38:57.550801542 (497h9m57.421884577s) [upload=436344]: Starting upload
Dec 26 08:38:57.550854924 (497h9m57.421937959s) [upload=436344]: Done starting upload
Dec 26 08:38:57.550948282 (497h9m57.422031337s) [upload=436344]: Uploading segments...
Dec 26 08:38:57.551041589 (497h9m57.422124634s) [upload=436344]: Got next segment. Inline: false
Dec 26 08:38:57.551075935 (497h9m57.42215897s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:38:57.551080012 (497h9m57.422163047s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:38:57.794454710 (497h9m57.665537765s) [seg_pos={0 0}, upload=436344]: Begin upload segment...
Dec 26 08:38:57.794524353 (497h9m57.665607418s) [seg_pos={0 0}, upload=436344]: Done begin upload segment.
Dec 26 08:38:57.794576082 (497h9m57.665659147s) [noise=true , address=45.91.116.9:4264 , piece_id=2ORB6O7KE77GUQWHCW6Y7DHSLBTBLM4TKFLODOIVVYW25Q5OBWPA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.797630828 (497h9m57.668713863s) [noise=true , address=145.79.227.49:4339 , piece_id=KRVOUKFLSJ3KLW36LC7AZJSIDX6R2DIFYCO3XBR72HTHYMPS3ZBA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.797865585 (497h9m57.66894863s) [noise=true , address=145.79.227.34:4907 , piece_id=GA5JH2LY2A3VYJKWMCHE2D3IMBAAV4FTDAGC4MKXSFN3NVW2MN3A, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.798672281 (497h9m57.669755326s) [noise=true , address=77.90.31.11:4556 , piece_id=YS5SN65I4IBKJSZQ4J3VRR3GGKFMJFPXNID2MJO3QOB3MBWB3ICA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.799110165 (497h9m57.67019321s) [noise=true , address=145.79.227.41:4085 , piece_id=VJQ6DHU4I3WH6FSBBYMEPDL7HEDKYHLPUPWYZMQ757KUUYVJVKNQ, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.799740235 (497h9m57.67082327s) [noise=true , address=45.91.116.13:4156 , piece_id=F2JUFZV3C43BUJTI2MOXC5GTPAQEEQKTNHGBXSLPBYKZBKIMBUVQ, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.800228084 (497h9m57.671311119s) [noise=true , address=145.79.227.11:4732 , piece_id=SLJSILIMRECVMG2TU2D5ERNAD5IHB4L6P7OGGOCB2L74NP4GBUYA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.800672631 (497h9m57.671755676s) [noise=true , address=77.90.31.21:4880 , piece_id=HF52OKJB4VMTMWSFPGSKY2VNROKY3KQAWX7F5SLNUTW4RN4FQQKQ, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.800974224 (497h9m57.672057269s) [noise=true , address=45.91.116.5:4300 , piece_id=N3ISDBB4Y3CFYKASZF746UOPKFVLURTL5N56ONA5FKPK6EBXNT5Q, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.801147795 (497h9m57.67223084s) [noise=true , address=145.79.227.45:4230 , piece_id=BUD6CB6MSRFE57Q7VKPJNPAQVZJLJCGZET33YBIW5RM7SRAYAXVA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.801647317 (497h9m57.672730362s) [noise=true , address=145.79.227.44:4194 , piece_id=JRMYPQ544IR2SFONYGQTP6G4EBVKFBQO3Z2UV4HIQYW3NF5NGBRA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.802178608 (497h9m57.673261653s) [noise=true , address=45.91.116.4:4939 , piece_id=6IFFEY2WIJXBVBNCSHKHIIOCXRMGHL7JVQF2W6Q6Y5KKFCSFDV6A, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.802800392 (497h9m57.673883437s) [noise=true , address=45.91.116.11:4120 , piece_id=R5H4WYA6OGWUWNTPZQMI7XVVRATVMKCKLEI2L2JVSSK5F773RT6A, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.803320633 (497h9m57.674403678s) [noise=true , address=45.91.116.7:4010 , piece_id=4J3MFEVEIFR6SOOH6UEHUV6RG32B6QTVTX4GSXBTDZWUZNXIQHMA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.803812630 (497h9m57.674895675s) [noise=true , address=145.79.227.5:4222 , piece_id=IOPZMUTM4YM7Y67H2AW262Z2BCDLGP5UV5FYSUIL3L2WZFH7XOHA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.805261679 (497h9m57.676344724s) [noise=true , address=45.91.116.8:4048 , piece_id=DATLF3Z2NGF3AHXNJM6PA3VAG4PWTIOM4ZXVHCWIVTSEGAJDNOPA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:59.437155680 (497h9m59.308238715s) [upload=436344]: Next returned nil segment
Dec 26 08:38:59.437168444 (497h9m59.308251479s) [upload=436344]: Waiting for error group managing segments...
Dec 26 08:38:59.816460292 (497h9m59.687543327s) [noise=true , address=145.79.227.5:4222 , piece_id=IOPZMUTM4YM7Y67H2AW262Z2BCDLGP5UV5FYSUIL3L2WZFH7XOHA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.824916334 (497h9m59.695999369s) [noise=true , address=145.79.227.41:4085 , piece_id=VJQ6DHU4I3WH6FSBBYMEPDL7HEDKYHLPUPWYZMQ757KUUYVJVKNQ, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.826964394 (497h9m59.698047439s) [noise=true , address=145.79.227.45:4230 , piece_id=BUD6CB6MSRFE57Q7VKPJNPAQVZJLJCGZET33YBIW5RM7SRAYAXVA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.837098581 (497h9m59.708181616s) [noise=true , address=145.79.227.11:4732 , piece_id=SLJSILIMRECVMG2TU2D5ERNAD5IHB4L6P7OGGOCB2L74NP4GBUYA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.842604505 (497h9m59.71368755s) [noise=true , address=145.79.227.49:4339 , piece_id=KRVOUKFLSJ3KLW36LC7AZJSIDX6R2DIFYCO3XBR72HTHYMPS3ZBA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.027048940 (497h9m59.898131985s) [noise=true , address=45.91.116.4:4939 , piece_id=6IFFEY2WIJXBVBNCSHKHIIOCXRMGHL7JVQF2W6Q6Y5KKFCSFDV6A, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.040774505 (497h9m59.91185755s) [noise=true , address=45.91.116.8:4048 , piece_id=DATLF3Z2NGF3AHXNJM6PA3VAG4PWTIOM4ZXVHCWIVTSEGAJDNOPA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.122985880 (497h9m59.994068925s) [noise=true , address=145.79.227.34:4907 , piece_id=GA5JH2LY2A3VYJKWMCHE2D3IMBAAV4FTDAGC4MKXSFN3NVW2MN3A, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.287166329 (497h10m0.158249364s) [noise=true , address=45.91.116.11:4120 , piece_id=R5H4WYA6OGWUWNTPZQMI7XVVRATVMKCKLEI2L2JVSSK5F773RT6A, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.768015600 (497h10m0.639098645s) [noise=true , address=45.91.116.13:4156 , piece_id=F2JUFZV3C43BUJTI2MOXC5GTPAQEEQKTNHGBXSLPBYKZBKIMBUVQ, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.783331975 (497h10m0.65441506s) [noise=true , address=45.91.116.9:4264 , piece_id=2ORB6O7KE77GUQWHCW6Y7DHSLBTBLM4TKFLODOIVVYW25Q5OBWPA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.880969624 (497h10m0.752052659s) [noise=true , address=45.91.116.7:4010 , piece_id=4J3MFEVEIFR6SOOH6UEHUV6RG32B6QTVTX4GSXBTDZWUZNXIQHMA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.890935490 (497h10m0.762018525s) [noise=true , address=77.90.31.21:4880 , piece_id=HF52OKJB4VMTMWSFPGSKY2VNROKY3KQAWX7F5SLNUTW4RN4FQQKQ, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.890963404 (497h10m0.762046449s) [seg_pos={0 0}, upload=436344]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:39:00.891553998 (497h10m0.762637043s) [noise=true , address=45.91.116.5:4300 , piece_id=N3ISDBB4Y3CFYKASZF746UOPKFVLURTL5N56ONA5FKPK6EBXNT5Q, seg_pos={0 0}, upload=436344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:39:00.891565530 (497h10m0.762648575s) [noise=true , address=45.91.116.5:4300 , piece_id=N3ISDBB4Y3CFYKASZF746UOPKFVLURTL5N56ONA5FKPK6EBXNT5Q, seg_pos={0 0}, upload=436344]: Long tail cancellation due to optimal threshold
Dec 26 08:39:00.891920796 (497h10m0.763003841s) [noise=true , address=77.90.31.11:4556 , piece_id=YS5SN65I4IBKJSZQ4J3VRR3GGKFMJFPXNID2MJO3QOB3MBWB3ICA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 26 08:39:00.891928662 (497h10m0.763011697s) [noise=true , address=77.90.31.11:4556 , piece_id=YS5SN65I4IBKJSZQ4J3VRR3GGKFMJFPXNID2MJO3QOB3MBWB3ICA, seg_pos={0 0}, upload=436344]: Long tail cancellation due to optimal threshold
Dec 26 08:39:00.892184439 (497h10m0.763267484s) [noise=true , address=145.79.227.44:4194 , piece_id=JRMYPQ544IR2SFONYGQTP6G4EBVKFBQO3Z2UV4HIQYW3NF5NGBRA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 26 08:39:00.892192624 (497h10m0.763275669s) [noise=true , address=145.79.227.44:4194 , piece_id=JRMYPQ544IR2SFONYGQTP6G4EBVKFBQO3Z2UV4HIQYW3NF5NGBRA, seg_pos={0 0}, upload=436344]: Long tail cancellation due to optimal threshold
Dec 26 08:39:00.892211019 (497h10m0.763294055s) [seg_pos={0 0}, upload=436344]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:39:00.892284519 (497h10m0.763367554s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:39:00.892290661 (497h10m0.763373696s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:39:01.185590685 (497h10m1.05667373s) [upload=436344]: Done uploading segments...
Dec 26 08:39:01.185617947 (497h10m1.056700992s) [upload=436344]: Upload finished. err: <nil>
Duration: 5.946239631s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
6.69997803s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:39:54 UTC)
orbiter warning: context deadline exceeded
Duration: 6.700009169s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
30.22450659s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:39:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 30.224523643s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................X........................................] duration:
3.510580029s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:39:30 UTC)
orbiter warning: context deadline exceeded
Duration: 3.510604034s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..............X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X] duration:
30.004370061s trace:
https://jaeger.dev.storj.io/trace/2ebe8a728d6ae142
FAIL (2025-12-26 08:39:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: DATLF3Z2NGF3AHXNJM6PA3VAG4PWTIOM4ZXVHCWIVTSEGAJDNOPA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: F2JUFZV3C43BUJTI2MOXC5GTPAQEEQKTNHGBXSLPBYKZBKIMBUVQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: HF52OKJB4VMTMWSFPGSKY2VNROKY3KQAWX7F5SLNUTW4RN4FQQKQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: R5H4WYA6OGWUWNTPZQMI7XVVRATVMKCKLEI2L2JVSSK5F773RT6A) context deadline exceeded
======LOGS======
Dec 26 08:38:57.550801542 (497h9m57.421884577s) [upload=436344]: Starting upload
Dec 26 08:38:57.550854924 (497h9m57.421937959s) [upload=436344]: Done starting upload
Dec 26 08:38:57.550948282 (497h9m57.422031337s) [upload=436344]: Uploading segments...
Dec 26 08:38:57.551041589 (497h9m57.422124634s) [upload=436344]: Got next segment. Inline: false
Dec 26 08:38:57.551075935 (497h9m57.42215897s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:38:57.551080012 (497h9m57.422163047s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:38:57.794454710 (497h9m57.665537765s) [seg_pos={0 0}, upload=436344]: Begin upload segment...
Dec 26 08:38:57.794524353 (497h9m57.665607418s) [seg_pos={0 0}, upload=436344]: Done begin upload segment.
Dec 26 08:38:57.794576082 (497h9m57.665659147s) [noise=true , address=45.91.116.9:4264 , piece_id=2ORB6O7KE77GUQWHCW6Y7DHSLBTBLM4TKFLODOIVVYW25Q5OBWPA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.797630828 (497h9m57.668713863s) [noise=true , address=145.79.227.49:4339 , piece_id=KRVOUKFLSJ3KLW36LC7AZJSIDX6R2DIFYCO3XBR72HTHYMPS3ZBA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.797865585 (497h9m57.66894863s) [noise=true , address=145.79.227.34:4907 , piece_id=GA5JH2LY2A3VYJKWMCHE2D3IMBAAV4FTDAGC4MKXSFN3NVW2MN3A, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.798672281 (497h9m57.669755326s) [noise=true , address=77.90.31.11:4556 , piece_id=YS5SN65I4IBKJSZQ4J3VRR3GGKFMJFPXNID2MJO3QOB3MBWB3ICA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.799110165 (497h9m57.67019321s) [noise=true , address=145.79.227.41:4085 , piece_id=VJQ6DHU4I3WH6FSBBYMEPDL7HEDKYHLPUPWYZMQ757KUUYVJVKNQ, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.799740235 (497h9m57.67082327s) [noise=true , address=45.91.116.13:4156 , piece_id=F2JUFZV3C43BUJTI2MOXC5GTPAQEEQKTNHGBXSLPBYKZBKIMBUVQ, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.800228084 (497h9m57.671311119s) [noise=true , address=145.79.227.11:4732 , piece_id=SLJSILIMRECVMG2TU2D5ERNAD5IHB4L6P7OGGOCB2L74NP4GBUYA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.800672631 (497h9m57.671755676s) [noise=true , address=77.90.31.21:4880 , piece_id=HF52OKJB4VMTMWSFPGSKY2VNROKY3KQAWX7F5SLNUTW4RN4FQQKQ, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.800974224 (497h9m57.672057269s) [noise=true , address=45.91.116.5:4300 , piece_id=N3ISDBB4Y3CFYKASZF746UOPKFVLURTL5N56ONA5FKPK6EBXNT5Q, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.801147795 (497h9m57.67223084s) [noise=true , address=145.79.227.45:4230 , piece_id=BUD6CB6MSRFE57Q7VKPJNPAQVZJLJCGZET33YBIW5RM7SRAYAXVA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.801647317 (497h9m57.672730362s) [noise=true , address=145.79.227.44:4194 , piece_id=JRMYPQ544IR2SFONYGQTP6G4EBVKFBQO3Z2UV4HIQYW3NF5NGBRA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.802178608 (497h9m57.673261653s) [noise=true , address=45.91.116.4:4939 , piece_id=6IFFEY2WIJXBVBNCSHKHIIOCXRMGHL7JVQF2W6Q6Y5KKFCSFDV6A, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.802800392 (497h9m57.673883437s) [noise=true , address=45.91.116.11:4120 , piece_id=R5H4WYA6OGWUWNTPZQMI7XVVRATVMKCKLEI2L2JVSSK5F773RT6A, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.803320633 (497h9m57.674403678s) [noise=true , address=45.91.116.7:4010 , piece_id=4J3MFEVEIFR6SOOH6UEHUV6RG32B6QTVTX4GSXBTDZWUZNXIQHMA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.803812630 (497h9m57.674895675s) [noise=true , address=145.79.227.5:4222 , piece_id=IOPZMUTM4YM7Y67H2AW262Z2BCDLGP5UV5FYSUIL3L2WZFH7XOHA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:57.805261679 (497h9m57.676344724s) [noise=true , address=45.91.116.8:4048 , piece_id=DATLF3Z2NGF3AHXNJM6PA3VAG4PWTIOM4ZXVHCWIVTSEGAJDNOPA, seg_pos={0 0}, upload=436344]: Uploading piece...
Dec 26 08:38:59.437155680 (497h9m59.308238715s) [upload=436344]: Next returned nil segment
Dec 26 08:38:59.437168444 (497h9m59.308251479s) [upload=436344]: Waiting for error group managing segments...
Dec 26 08:38:59.816460292 (497h9m59.687543327s) [noise=true , address=145.79.227.5:4222 , piece_id=IOPZMUTM4YM7Y67H2AW262Z2BCDLGP5UV5FYSUIL3L2WZFH7XOHA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.824916334 (497h9m59.695999369s) [noise=true , address=145.79.227.41:4085 , piece_id=VJQ6DHU4I3WH6FSBBYMEPDL7HEDKYHLPUPWYZMQ757KUUYVJVKNQ, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.826964394 (497h9m59.698047439s) [noise=true , address=145.79.227.45:4230 , piece_id=BUD6CB6MSRFE57Q7VKPJNPAQVZJLJCGZET33YBIW5RM7SRAYAXVA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.837098581 (497h9m59.708181616s) [noise=true , address=145.79.227.11:4732 , piece_id=SLJSILIMRECVMG2TU2D5ERNAD5IHB4L6P7OGGOCB2L74NP4GBUYA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:38:59.842604505 (497h9m59.71368755s) [noise=true , address=145.79.227.49:4339 , piece_id=KRVOUKFLSJ3KLW36LC7AZJSIDX6R2DIFYCO3XBR72HTHYMPS3ZBA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.027048940 (497h9m59.898131985s) [noise=true , address=45.91.116.4:4939 , piece_id=6IFFEY2WIJXBVBNCSHKHIIOCXRMGHL7JVQF2W6Q6Y5KKFCSFDV6A, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.040774505 (497h9m59.91185755s) [noise=true , address=45.91.116.8:4048 , piece_id=DATLF3Z2NGF3AHXNJM6PA3VAG4PWTIOM4ZXVHCWIVTSEGAJDNOPA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.122985880 (497h9m59.994068925s) [noise=true , address=145.79.227.34:4907 , piece_id=GA5JH2LY2A3VYJKWMCHE2D3IMBAAV4FTDAGC4MKXSFN3NVW2MN3A, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.287166329 (497h10m0.158249364s) [noise=true , address=45.91.116.11:4120 , piece_id=R5H4WYA6OGWUWNTPZQMI7XVVRATVMKCKLEI2L2JVSSK5F773RT6A, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.768015600 (497h10m0.639098645s) [noise=true , address=45.91.116.13:4156 , piece_id=F2JUFZV3C43BUJTI2MOXC5GTPAQEEQKTNHGBXSLPBYKZBKIMBUVQ, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.783331975 (497h10m0.65441506s) [noise=true , address=45.91.116.9:4264 , piece_id=2ORB6O7KE77GUQWHCW6Y7DHSLBTBLM4TKFLODOIVVYW25Q5OBWPA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.880969624 (497h10m0.752052659s) [noise=true , address=45.91.116.7:4010 , piece_id=4J3MFEVEIFR6SOOH6UEHUV6RG32B6QTVTX4GSXBTDZWUZNXIQHMA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.890935490 (497h10m0.762018525s) [noise=true , address=77.90.31.21:4880 , piece_id=HF52OKJB4VMTMWSFPGSKY2VNROKY3KQAWX7F5SLNUTW4RN4FQQKQ, seg_pos={0 0}, upload=436344]: Done uploading piece. err: <nil>
Dec 26 08:39:00.890963404 (497h10m0.762046449s) [seg_pos={0 0}, upload=436344]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:39:00.891553998 (497h10m0.762637043s) [noise=true , address=45.91.116.5:4300 , piece_id=N3ISDBB4Y3CFYKASZF746UOPKFVLURTL5N56ONA5FKPK6EBXNT5Q, seg_pos={0 0}, upload=436344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:39:00.891565530 (497h10m0.762648575s) [noise=true , address=45.91.116.5:4300 , piece_id=N3ISDBB4Y3CFYKASZF746UOPKFVLURTL5N56ONA5FKPK6EBXNT5Q, seg_pos={0 0}, upload=436344]: Long tail cancellation due to optimal threshold
Dec 26 08:39:00.891920796 (497h10m0.763003841s) [noise=true , address=77.90.31.11:4556 , piece_id=YS5SN65I4IBKJSZQ4J3VRR3GGKFMJFPXNID2MJO3QOB3MBWB3ICA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 26 08:39:00.891928662 (497h10m0.763011697s) [noise=true , address=77.90.31.11:4556 , piece_id=YS5SN65I4IBKJSZQ4J3VRR3GGKFMJFPXNID2MJO3QOB3MBWB3ICA, seg_pos={0 0}, upload=436344]: Long tail cancellation due to optimal threshold
Dec 26 08:39:00.892184439 (497h10m0.763267484s) [noise=true , address=145.79.227.44:4194 , piece_id=JRMYPQ544IR2SFONYGQTP6G4EBVKFBQO3Z2UV4HIQYW3NF5NGBRA, seg_pos={0 0}, upload=436344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 26 08:39:00.892192624 (497h10m0.763275669s) [noise=true , address=145.79.227.44:4194 , piece_id=JRMYPQ544IR2SFONYGQTP6G4EBVKFBQO3Z2UV4HIQYW3NF5NGBRA, seg_pos={0 0}, upload=436344]: Long tail cancellation due to optimal threshold
Dec 26 08:39:00.892211019 (497h10m0.763294055s) [seg_pos={0 0}, upload=436344]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:39:00.892284519 (497h10m0.763367554s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:39:00.892290661 (497h10m0.763373696s) [upload=436344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:39:01.185590685 (497h10m1.05667373s) [upload=436344]: Done uploading segments...
Dec 26 08:39:01.185617947 (497h10m1.056700992s) [upload=436344]: Upload finished. err: <nil>
Duration: 30.004392134s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
36.988693545s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:39:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 36.988711137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.859595253s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:39:11 UTC)
orbiter warning: context deadline exceeded
Duration: 5.859636732s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...............X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..] duration:
5.79572829s trace:
https://jaeger.dev.storj.io/trace/3ab67d40e43639e2
FAIL (2025-12-26 08:38:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A) context deadline exceeded
======LOGS======
Dec 26 08:36:57.111389744 (497h7m56.982472779s) [upload=436316]: Starting upload
Dec 26 08:36:57.111449608 (497h7m56.982532644s) [upload=436316]: Done starting upload
Dec 26 08:36:57.111584556 (497h7m56.982667601s) [upload=436316]: Uploading segments...
Dec 26 08:36:57.111722809 (497h7m56.982805854s) [upload=436316]: Got next segment. Inline: false
Dec 26 08:36:57.111769507 (497h7m56.982852542s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:36:57.111774637 (497h7m56.982857672s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:36:57.354083156 (497h7m57.225166191s) [seg_pos={0 0}, upload=436316]: Begin upload segment...
Dec 26 08:36:57.354148629 (497h7m57.225231664s) [seg_pos={0 0}, upload=436316]: Done begin upload segment.
Dec 26 08:36:57.354174649 (497h7m57.225257684s) [noise=true , address=145.79.227.38:4978 , piece_id=URJFEZ6E5Y64WNCTHLHUHFNV7NCI6T6P7W6LJP5UVPYEALLGXAOA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.354361245 (497h7m57.22544429s) [noise=true , address=145.79.227.11:4730 , piece_id=JIH6F3YFXBAPUM5D4OQY6HVSYIRSSU5JZSF3BNY3256L5SSN476A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355021893 (497h7m57.226104928s) [noise=true , address=45.91.116.7:4012 , piece_id=Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355310232 (497h7m57.226393267s) [noise=true , address=145.79.227.37:4567 , piece_id=J6C3EVSAO2F5AMVG37TMXKNSR62R5HJ2CEVA2CMV2UGUGJEZLJXQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355559697 (497h7m57.226642742s) [noise=true , address=77.90.31.14:4664 , piece_id=EYQTTU6YCZNAWDJKEJVJFX7WNAVWMLQYJY667LFSTMJSQXSCANEA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355854538 (497h7m57.226937583s) [noise=true , address=145.79.227.15:4875 , piece_id=IYOFVGPGGSGHXLXEVN2ON45VFPERFWTCEZJ3KKFQQGEJ6JII4KFQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355998092 (497h7m57.227081137s) [noise=true , address=45.91.116.9:4263 , piece_id=BCMLAJV2W7AMV6MO7ELZLD2S533EWHG3XEYW2GWBVR2UV2UA57RA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356093593 (497h7m57.227176638s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356386692 (497h7m57.227469727s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356687253 (497h7m57.227770288s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356723983 (497h7m57.227807028s) [noise=true , address=145.79.227.32:4834 , piece_id=OPJ64IOCUMHHSX36LSXZPDZ64INN7DO7FYUVKDJBNTVHW3GFIJ5Q, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356980332 (497h7m57.228063367s) [noise=true , address=145.79.227.6:4259 , piece_id=MRFU2KVOVVSOLWCYNTA7VBFKKC6ZMP7PQ47N3R64UI6I4255YU5A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.357346569 (497h7m57.228429604s) [noise=true , address=145.79.227.35:4482 , piece_id=Q66XOGAYSYHBG6K7Q2GEBEXQ7QCU7QWAYZYIIIAYF26NRTNMEYNA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358089444 (497h7m57.229172489s) [noise=true , address=77.90.31.21:4880 , piece_id=LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358599815 (497h7m57.22968286s) [noise=true , address=145.79.227.19:4078 , piece_id=J6K6W262IMKQLMIRRC55YA4HFEZVZIS4O4ZGRDPRQ4AED6KH4NEA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358934924 (497h7m57.230017959s) [noise=true , address=145.79.227.51:4409 , piece_id=BINIAAK2X6AWP6MEQA567QKYCM4L3QZPVAXZOLQXAL4LTGDH7OCQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:59.060417033 (497h7m58.931500068s) [upload=436316]: Next returned nil segment
Dec 26 08:36:59.060450427 (497h7m58.931533462s) [upload=436316]: Waiting for error group managing segments...
Dec 26 08:36:59.358877915 (497h7m59.22996096s) [noise=true , address=145.79.227.11:4730 , piece_id=JIH6F3YFXBAPUM5D4OQY6HVSYIRSSU5JZSF3BNY3256L5SSN476A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.360804302 (497h7m59.231887337s) [noise=true , address=145.79.227.6:4259 , piece_id=MRFU2KVOVVSOLWCYNTA7VBFKKC6ZMP7PQ47N3R64UI6I4255YU5A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.364244353 (497h7m59.235327388s) [noise=true , address=145.79.227.37:4567 , piece_id=J6C3EVSAO2F5AMVG37TMXKNSR62R5HJ2CEVA2CMV2UGUGJEZLJXQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.366999930 (497h7m59.238082975s) [noise=true , address=145.79.227.19:4078 , piece_id=J6K6W262IMKQLMIRRC55YA4HFEZVZIS4O4ZGRDPRQ4AED6KH4NEA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.367982963 (497h7m59.239066008s) [noise=true , address=145.79.227.35:4482 , piece_id=Q66XOGAYSYHBG6K7Q2GEBEXQ7QCU7QWAYZYIIIAYF26NRTNMEYNA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.375436826 (497h7m59.246519861s) [noise=true , address=145.79.227.51:4409 , piece_id=BINIAAK2X6AWP6MEQA567QKYCM4L3QZPVAXZOLQXAL4LTGDH7OCQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.378587835 (497h7m59.249670871s) [noise=true , address=145.79.227.15:4875 , piece_id=IYOFVGPGGSGHXLXEVN2ON45VFPERFWTCEZJ3KKFQQGEJ6JII4KFQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.379344897 (497h7m59.250427942s) [noise=true , address=145.79.227.32:4834 , piece_id=OPJ64IOCUMHHSX36LSXZPDZ64INN7DO7FYUVKDJBNTVHW3GFIJ5Q, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.382856735 (497h7m59.25393977s) [noise=true , address=145.79.227.38:4978 , piece_id=URJFEZ6E5Y64WNCTHLHUHFNV7NCI6T6P7W6LJP5UVPYEALLGXAOA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.526539331 (497h7m59.397622377s) [noise=true , address=77.90.31.14:4664 , piece_id=EYQTTU6YCZNAWDJKEJVJFX7WNAVWMLQYJY667LFSTMJSQXSCANEA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.577254883 (497h7m59.448337918s) [noise=true , address=45.91.116.9:4263 , piece_id=BCMLAJV2W7AMV6MO7ELZLD2S533EWHG3XEYW2GWBVR2UV2UA57RA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.591548961 (497h7m59.462632006s) [noise=true , address=45.91.116.7:4012 , piece_id=Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.619048514 (497h7m59.49013155s) [noise=true , address=77.90.31.21:4880 , piece_id=LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.619070547 (497h7m59.490153592s) [seg_pos={0 0}, upload=436316]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:36:59.619375418 (497h7m59.490458463s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:59.619382431 (497h7m59.490465466s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619500345 (497h7m59.49058338s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:59.619506097 (497h7m59.490589142s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619672392 (497h7m59.490755428s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:52624->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:36:59.619678043 (497h7m59.490761078s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619692181 (497h7m59.490775216s) [seg_pos={0 0}, upload=436316]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:36:59.619776982 (497h7m59.490860017s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:36:59.619783043 (497h7m59.490866078s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:36:59.928582953 (497h7m59.799665988s) [upload=436316]: Done uploading segments...
Dec 26 08:36:59.928602930 (497h7m59.799685965s) [upload=436316]: Upload finished. err: <nil>
Duration: 5.795757905s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
53.241073774s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:38:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 53.24109278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................X.......................................] duration:
3.461066838s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:38:24 UTC)
orbiter warning: context deadline exceeded
Duration: 3.461096884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
8.547741595s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:38:06 UTC)
orbiter warning: context deadline exceeded
Duration: 8.547765309s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
54.725861479s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:37:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 54.725879925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
3.598800375s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:37:36 UTC)
orbiter warning: context deadline exceeded
Duration: 3.598830903s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [................X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX.] duration:
5.917031842s trace:
https://jaeger.dev.storj.io/trace/3ab67d40e43639e2
FAIL (2025-12-26 08:37:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A) context deadline exceeded
======LOGS======
Dec 26 08:36:57.111389744 (497h7m56.982472779s) [upload=436316]: Starting upload
Dec 26 08:36:57.111449608 (497h7m56.982532644s) [upload=436316]: Done starting upload
Dec 26 08:36:57.111584556 (497h7m56.982667601s) [upload=436316]: Uploading segments...
Dec 26 08:36:57.111722809 (497h7m56.982805854s) [upload=436316]: Got next segment. Inline: false
Dec 26 08:36:57.111769507 (497h7m56.982852542s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:36:57.111774637 (497h7m56.982857672s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:36:57.354083156 (497h7m57.225166191s) [seg_pos={0 0}, upload=436316]: Begin upload segment...
Dec 26 08:36:57.354148629 (497h7m57.225231664s) [seg_pos={0 0}, upload=436316]: Done begin upload segment.
Dec 26 08:36:57.354174649 (497h7m57.225257684s) [noise=true , address=145.79.227.38:4978 , piece_id=URJFEZ6E5Y64WNCTHLHUHFNV7NCI6T6P7W6LJP5UVPYEALLGXAOA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.354361245 (497h7m57.22544429s) [noise=true , address=145.79.227.11:4730 , piece_id=JIH6F3YFXBAPUM5D4OQY6HVSYIRSSU5JZSF3BNY3256L5SSN476A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355021893 (497h7m57.226104928s) [noise=true , address=45.91.116.7:4012 , piece_id=Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355310232 (497h7m57.226393267s) [noise=true , address=145.79.227.37:4567 , piece_id=J6C3EVSAO2F5AMVG37TMXKNSR62R5HJ2CEVA2CMV2UGUGJEZLJXQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355559697 (497h7m57.226642742s) [noise=true , address=77.90.31.14:4664 , piece_id=EYQTTU6YCZNAWDJKEJVJFX7WNAVWMLQYJY667LFSTMJSQXSCANEA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355854538 (497h7m57.226937583s) [noise=true , address=145.79.227.15:4875 , piece_id=IYOFVGPGGSGHXLXEVN2ON45VFPERFWTCEZJ3KKFQQGEJ6JII4KFQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355998092 (497h7m57.227081137s) [noise=true , address=45.91.116.9:4263 , piece_id=BCMLAJV2W7AMV6MO7ELZLD2S533EWHG3XEYW2GWBVR2UV2UA57RA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356093593 (497h7m57.227176638s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356386692 (497h7m57.227469727s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356687253 (497h7m57.227770288s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356723983 (497h7m57.227807028s) [noise=true , address=145.79.227.32:4834 , piece_id=OPJ64IOCUMHHSX36LSXZPDZ64INN7DO7FYUVKDJBNTVHW3GFIJ5Q, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356980332 (497h7m57.228063367s) [noise=true , address=145.79.227.6:4259 , piece_id=MRFU2KVOVVSOLWCYNTA7VBFKKC6ZMP7PQ47N3R64UI6I4255YU5A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.357346569 (497h7m57.228429604s) [noise=true , address=145.79.227.35:4482 , piece_id=Q66XOGAYSYHBG6K7Q2GEBEXQ7QCU7QWAYZYIIIAYF26NRTNMEYNA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358089444 (497h7m57.229172489s) [noise=true , address=77.90.31.21:4880 , piece_id=LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358599815 (497h7m57.22968286s) [noise=true , address=145.79.227.19:4078 , piece_id=J6K6W262IMKQLMIRRC55YA4HFEZVZIS4O4ZGRDPRQ4AED6KH4NEA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358934924 (497h7m57.230017959s) [noise=true , address=145.79.227.51:4409 , piece_id=BINIAAK2X6AWP6MEQA567QKYCM4L3QZPVAXZOLQXAL4LTGDH7OCQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:59.060417033 (497h7m58.931500068s) [upload=436316]: Next returned nil segment
Dec 26 08:36:59.060450427 (497h7m58.931533462s) [upload=436316]: Waiting for error group managing segments...
Dec 26 08:36:59.358877915 (497h7m59.22996096s) [noise=true , address=145.79.227.11:4730 , piece_id=JIH6F3YFXBAPUM5D4OQY6HVSYIRSSU5JZSF3BNY3256L5SSN476A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.360804302 (497h7m59.231887337s) [noise=true , address=145.79.227.6:4259 , piece_id=MRFU2KVOVVSOLWCYNTA7VBFKKC6ZMP7PQ47N3R64UI6I4255YU5A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.364244353 (497h7m59.235327388s) [noise=true , address=145.79.227.37:4567 , piece_id=J6C3EVSAO2F5AMVG37TMXKNSR62R5HJ2CEVA2CMV2UGUGJEZLJXQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.366999930 (497h7m59.238082975s) [noise=true , address=145.79.227.19:4078 , piece_id=J6K6W262IMKQLMIRRC55YA4HFEZVZIS4O4ZGRDPRQ4AED6KH4NEA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.367982963 (497h7m59.239066008s) [noise=true , address=145.79.227.35:4482 , piece_id=Q66XOGAYSYHBG6K7Q2GEBEXQ7QCU7QWAYZYIIIAYF26NRTNMEYNA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.375436826 (497h7m59.246519861s) [noise=true , address=145.79.227.51:4409 , piece_id=BINIAAK2X6AWP6MEQA567QKYCM4L3QZPVAXZOLQXAL4LTGDH7OCQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.378587835 (497h7m59.249670871s) [noise=true , address=145.79.227.15:4875 , piece_id=IYOFVGPGGSGHXLXEVN2ON45VFPERFWTCEZJ3KKFQQGEJ6JII4KFQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.379344897 (497h7m59.250427942s) [noise=true , address=145.79.227.32:4834 , piece_id=OPJ64IOCUMHHSX36LSXZPDZ64INN7DO7FYUVKDJBNTVHW3GFIJ5Q, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.382856735 (497h7m59.25393977s) [noise=true , address=145.79.227.38:4978 , piece_id=URJFEZ6E5Y64WNCTHLHUHFNV7NCI6T6P7W6LJP5UVPYEALLGXAOA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.526539331 (497h7m59.397622377s) [noise=true , address=77.90.31.14:4664 , piece_id=EYQTTU6YCZNAWDJKEJVJFX7WNAVWMLQYJY667LFSTMJSQXSCANEA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.577254883 (497h7m59.448337918s) [noise=true , address=45.91.116.9:4263 , piece_id=BCMLAJV2W7AMV6MO7ELZLD2S533EWHG3XEYW2GWBVR2UV2UA57RA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.591548961 (497h7m59.462632006s) [noise=true , address=45.91.116.7:4012 , piece_id=Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.619048514 (497h7m59.49013155s) [noise=true , address=77.90.31.21:4880 , piece_id=LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.619070547 (497h7m59.490153592s) [seg_pos={0 0}, upload=436316]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:36:59.619375418 (497h7m59.490458463s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:59.619382431 (497h7m59.490465466s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619500345 (497h7m59.49058338s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:59.619506097 (497h7m59.490589142s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619672392 (497h7m59.490755428s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:52624->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:36:59.619678043 (497h7m59.490761078s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619692181 (497h7m59.490775216s) [seg_pos={0 0}, upload=436316]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:36:59.619776982 (497h7m59.490860017s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:36:59.619783043 (497h7m59.490866078s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:36:59.928582953 (497h7m59.799665988s) [upload=436316]: Done uploading segments...
Dec 26 08:36:59.928602930 (497h7m59.799685965s) [upload=436316]: Upload finished. err: <nil>
Duration: 5.917047501s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.................X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX] duration:
30.003924043s trace:
https://jaeger.dev.storj.io/trace/3ab67d40e43639e2
FAIL (2025-12-26 08:37:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A) context deadline exceeded
======LOGS======
Dec 26 08:36:57.111389744 (497h7m56.982472779s) [upload=436316]: Starting upload
Dec 26 08:36:57.111449608 (497h7m56.982532644s) [upload=436316]: Done starting upload
Dec 26 08:36:57.111584556 (497h7m56.982667601s) [upload=436316]: Uploading segments...
Dec 26 08:36:57.111722809 (497h7m56.982805854s) [upload=436316]: Got next segment. Inline: false
Dec 26 08:36:57.111769507 (497h7m56.982852542s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:36:57.111774637 (497h7m56.982857672s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:36:57.354083156 (497h7m57.225166191s) [seg_pos={0 0}, upload=436316]: Begin upload segment...
Dec 26 08:36:57.354148629 (497h7m57.225231664s) [seg_pos={0 0}, upload=436316]: Done begin upload segment.
Dec 26 08:36:57.354174649 (497h7m57.225257684s) [noise=true , address=145.79.227.38:4978 , piece_id=URJFEZ6E5Y64WNCTHLHUHFNV7NCI6T6P7W6LJP5UVPYEALLGXAOA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.354361245 (497h7m57.22544429s) [noise=true , address=145.79.227.11:4730 , piece_id=JIH6F3YFXBAPUM5D4OQY6HVSYIRSSU5JZSF3BNY3256L5SSN476A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355021893 (497h7m57.226104928s) [noise=true , address=45.91.116.7:4012 , piece_id=Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355310232 (497h7m57.226393267s) [noise=true , address=145.79.227.37:4567 , piece_id=J6C3EVSAO2F5AMVG37TMXKNSR62R5HJ2CEVA2CMV2UGUGJEZLJXQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355559697 (497h7m57.226642742s) [noise=true , address=77.90.31.14:4664 , piece_id=EYQTTU6YCZNAWDJKEJVJFX7WNAVWMLQYJY667LFSTMJSQXSCANEA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355854538 (497h7m57.226937583s) [noise=true , address=145.79.227.15:4875 , piece_id=IYOFVGPGGSGHXLXEVN2ON45VFPERFWTCEZJ3KKFQQGEJ6JII4KFQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.355998092 (497h7m57.227081137s) [noise=true , address=45.91.116.9:4263 , piece_id=BCMLAJV2W7AMV6MO7ELZLD2S533EWHG3XEYW2GWBVR2UV2UA57RA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356093593 (497h7m57.227176638s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356386692 (497h7m57.227469727s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356687253 (497h7m57.227770288s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356723983 (497h7m57.227807028s) [noise=true , address=145.79.227.32:4834 , piece_id=OPJ64IOCUMHHSX36LSXZPDZ64INN7DO7FYUVKDJBNTVHW3GFIJ5Q, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.356980332 (497h7m57.228063367s) [noise=true , address=145.79.227.6:4259 , piece_id=MRFU2KVOVVSOLWCYNTA7VBFKKC6ZMP7PQ47N3R64UI6I4255YU5A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.357346569 (497h7m57.228429604s) [noise=true , address=145.79.227.35:4482 , piece_id=Q66XOGAYSYHBG6K7Q2GEBEXQ7QCU7QWAYZYIIIAYF26NRTNMEYNA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358089444 (497h7m57.229172489s) [noise=true , address=77.90.31.21:4880 , piece_id=LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358599815 (497h7m57.22968286s) [noise=true , address=145.79.227.19:4078 , piece_id=J6K6W262IMKQLMIRRC55YA4HFEZVZIS4O4ZGRDPRQ4AED6KH4NEA, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:57.358934924 (497h7m57.230017959s) [noise=true , address=145.79.227.51:4409 , piece_id=BINIAAK2X6AWP6MEQA567QKYCM4L3QZPVAXZOLQXAL4LTGDH7OCQ, seg_pos={0 0}, upload=436316]: Uploading piece...
Dec 26 08:36:59.060417033 (497h7m58.931500068s) [upload=436316]: Next returned nil segment
Dec 26 08:36:59.060450427 (497h7m58.931533462s) [upload=436316]: Waiting for error group managing segments...
Dec 26 08:36:59.358877915 (497h7m59.22996096s) [noise=true , address=145.79.227.11:4730 , piece_id=JIH6F3YFXBAPUM5D4OQY6HVSYIRSSU5JZSF3BNY3256L5SSN476A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.360804302 (497h7m59.231887337s) [noise=true , address=145.79.227.6:4259 , piece_id=MRFU2KVOVVSOLWCYNTA7VBFKKC6ZMP7PQ47N3R64UI6I4255YU5A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.364244353 (497h7m59.235327388s) [noise=true , address=145.79.227.37:4567 , piece_id=J6C3EVSAO2F5AMVG37TMXKNSR62R5HJ2CEVA2CMV2UGUGJEZLJXQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.366999930 (497h7m59.238082975s) [noise=true , address=145.79.227.19:4078 , piece_id=J6K6W262IMKQLMIRRC55YA4HFEZVZIS4O4ZGRDPRQ4AED6KH4NEA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.367982963 (497h7m59.239066008s) [noise=true , address=145.79.227.35:4482 , piece_id=Q66XOGAYSYHBG6K7Q2GEBEXQ7QCU7QWAYZYIIIAYF26NRTNMEYNA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.375436826 (497h7m59.246519861s) [noise=true , address=145.79.227.51:4409 , piece_id=BINIAAK2X6AWP6MEQA567QKYCM4L3QZPVAXZOLQXAL4LTGDH7OCQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.378587835 (497h7m59.249670871s) [noise=true , address=145.79.227.15:4875 , piece_id=IYOFVGPGGSGHXLXEVN2ON45VFPERFWTCEZJ3KKFQQGEJ6JII4KFQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.379344897 (497h7m59.250427942s) [noise=true , address=145.79.227.32:4834 , piece_id=OPJ64IOCUMHHSX36LSXZPDZ64INN7DO7FYUVKDJBNTVHW3GFIJ5Q, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.382856735 (497h7m59.25393977s) [noise=true , address=145.79.227.38:4978 , piece_id=URJFEZ6E5Y64WNCTHLHUHFNV7NCI6T6P7W6LJP5UVPYEALLGXAOA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.526539331 (497h7m59.397622377s) [noise=true , address=77.90.31.14:4664 , piece_id=EYQTTU6YCZNAWDJKEJVJFX7WNAVWMLQYJY667LFSTMJSQXSCANEA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.577254883 (497h7m59.448337918s) [noise=true , address=45.91.116.9:4263 , piece_id=BCMLAJV2W7AMV6MO7ELZLD2S533EWHG3XEYW2GWBVR2UV2UA57RA, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.591548961 (497h7m59.462632006s) [noise=true , address=45.91.116.7:4012 , piece_id=Y3Q7W6DXDKLPXCZTNQJGHUBN3PL3CHNDFSQZ4AZN72CPXONNHH7Q, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.619048514 (497h7m59.49013155s) [noise=true , address=77.90.31.21:4880 , piece_id=LLEIUAHOGK3VTL4PCEHKE7J6UD4LRLA65X74TP2C2A4AQM3SLQ4A, seg_pos={0 0}, upload=436316]: Done uploading piece. err: <nil>
Dec 26 08:36:59.619070547 (497h7m59.490153592s) [seg_pos={0 0}, upload=436316]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:36:59.619375418 (497h7m59.490458463s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:59.619382431 (497h7m59.490465466s) [noise=true , address=145.79.227.44:4195 , piece_id=RHZ6UWCEDMS64DPGLKGAPYI5LYDX2VOH25BBKHVVOSKMAU5ZYHEQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619500345 (497h7m59.49058338s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:59.619506097 (497h7m59.490589142s) [noise=true , address=45.91.116.10:4084 , piece_id=CPHML3J5E7YIHLMX6M5CTPDRACXOR2NQMWXAQIUVM46MRGSLFTHQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619672392 (497h7m59.490755428s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:52624->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:36:59.619678043 (497h7m59.490761078s) [noise=true , address=45.91.116.12:4226 , piece_id=6L5UXHCQZ7S7XTAT3S4C4UFKDTY7NVRBLUW637ULRV3E4LRHFOMQ, seg_pos={0 0}, upload=436316]: Long tail cancellation due to optimal threshold
Dec 26 08:36:59.619692181 (497h7m59.490775216s) [seg_pos={0 0}, upload=436316]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:36:59.619776982 (497h7m59.490860017s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:36:59.619783043 (497h7m59.490866078s) [upload=436316]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:36:59.928582953 (497h7m59.799665988s) [upload=436316]: Done uploading segments...
Dec 26 08:36:59.928602930 (497h7m59.799685965s) [upload=436316]: Upload finished. err: <nil>
Duration: 30.003962706s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
3.51556486s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:37:18 UTC)
orbiter warning: context deadline exceeded
Duration: 3.51559091s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
7.442653643s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:37:01 UTC)
orbiter warning: context deadline exceeded
Duration: 7.442692868s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..................X..X.XX.X.X..X..X.XXX.....X......X.X...X.X] duration:
30.004934076s trace:
https://jaeger.dev.storj.io/trace/11dac8edaefdbcf1
FAIL (2025-12-26 08:36:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: ZD4B23PRY4MXW4ZMSTNZEQ6SXHU4RYKQ7ZXGTTNGO5EAUGEWSCKQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: OPFFPWBCSSTJGSK6GUAHZYJ5WAUYVMPSX622VGVDBLKGS2LFAG7Q) context deadline exceeded
======LOGS======
(497h7m23.518764323s) [upload=436306]: Uploading segments...
Dec 26 08:36:23.647732324 (497h7m23.518815369s) [upload=436306]: Got next segment. Inline: false
Dec 26 08:36:23.647834519 (497h7m23.518917564s) [upload=436306]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:36:23.647844007 (497h7m23.518927052s) [upload=436306]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:36:23.902770832 (497h7m23.773853877s) [seg_pos={0 0}, upload=436306]: Begin upload segment...
Dec 26 08:36:23.902812742 (497h7m23.773895777s) [seg_pos={0 0}, upload=436306]: Done begin upload segment.
Dec 26 08:36:23.902838681 (497h7m23.773921716s) [noise=true , address=145.79.227.39:4014 , piece_id=PF64AYWNZQ6L777DMA752M6ICKHZ7Y2CFC27K63FJVJ4JI6VURTA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.902957547 (497h7m23.774040582s) [noise=true , address=45.91.116.10:4084 , piece_id=ZD4B23PRY4MXW4ZMSTNZEQ6SXHU4RYKQ7ZXGTTNGO5EAUGEWSCKQ, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.903494430 (497h7m23.774577475s) [noise=true , address=77.90.31.14:4664 , piece_id=OPFFPWBCSSTJGSK6GUAHZYJ5WAUYVMPSX622VGVDBLKGS2LFAG7Q, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.903850859 (497h7m23.774933894s) [noise=true , address=145.79.227.12:4767 , piece_id=R5GHKRYCB4YMZKCOOHNJKXCLWNCUENLS4R5NU5CG7HXDK5NPEG3Q, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.904152303 (497h7m23.775235338s) [noise=true , address=145.79.227.45:4231 , piece_id=VKW7CDXBYQOB4MYE3BVFWME4M4N7G63YM5M57JNK2UYOMHXTLRCA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.904429551 (497h7m23.775512586s) [noise=true , address=45.91.116.8:4046 , piece_id=QJ55D3XIQARE6UABB2RH4N7M2VRI7OFWI3FHZJNZCHKA2SJMNTCQ, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.904748318 (497h7m23.775831353s) [noise=true , address=145.79.227.14:4837 , piece_id=ADWB7Z7GVBVWVFWX7NGZ4G4TGWA7JKFV53TUSILYVHJ5VD46H7FQ, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.905050022 (497h7m23.776133067s) [noise=true , address=45.91.116.14:4189 , piece_id=ZMVDHOUM3QGGGOUWNFOM6IGUJJTQ2LIVWTYVT3GKYMYCY3TSKJBQ, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.905402814 (497h7m23.776485849s) [noise=true , address=145.79.227.13:4804 , piece_id=NSKUIQ5CU4JACTTEAQOQGMCP35CKWSRQ5LZOXLNZUDDTQU5JFFVA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.906916656 (497h7m23.777999692s) [noise=true , address=45.91.116.12:4226 , piece_id=RW7MJTDV2PT2JAXIZ3IMN7LXSSSRYOXTJWNKMGLQWGC4K42KUGTA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.907200637 (497h7m23.778283672s) [noise=true , address=77.90.31.10:4519 , piece_id=53F4BGZNBVXO3522TDUDBYAUY36YTDPD6GNDQEHMNS6SJS3Z3GHQ, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.908551750 (497h7m23.779634785s) [noise=true , address=145.79.227.48:4302 , piece_id=VJ7DDH6P5KFTQYYOXIH654Z4CYDR3GFIHPSHALHM7A2XNJMXM3FA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.910063148 (497h7m23.781146183s) [noise=true , address=77.90.31.13:4629 , piece_id=2M2RHAGORCTRJ2IBMANNXFE4SCYRCAM6JECRPYMDA5Q2DK3LWKMA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.911632437 (497h7m23.782715482s) [noise=true , address=77.90.31.17:4773 , piece_id=ZJR3RHK7FJUTHA4HTRTVVAFAAK6GCR3KPU5SI35QGIWRSMHNMOWA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.913079962 (497h7m23.784163007s) [noise=true , address=145.79.227.50:4373 , piece_id=AFFA6ETOV54HUSEAP7J7PAQ3TYUK6NLPMK2V333PKB5MXFTB6RLQ, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:23.915028573 (497h7m23.786111608s) [noise=true , address=145.79.227.51:4411 , piece_id=5KFDDQI2CTAJHZA6C3L5TE5YONMMUGHJJS277DEKFHIU6XNOHSSA, seg_pos={0 0}, upload=436306]: Uploading piece...
Dec 26 08:36:25.637061615 (497h7m25.50814465s) [upload=436306]: Next returned nil segment
Dec 26 08:36:25.637089929 (497h7m25.508172974s) [upload=436306]: Waiting for error group managing segments...
Dec 26 08:36:25.914575740 (497h7m25.785658785s) [noise=true , address=145.79.227.14:4837 , piece_id=ADWB7Z7GVBVWVFWX7NGZ4G4TGWA7JKFV53TUSILYVHJ5VD46H7FQ, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.926391249 (497h7m25.797474314s) [noise=true , address=145.79.227.45:4231 , piece_id=VKW7CDXBYQOB4MYE3BVFWME4M4N7G63YM5M57JNK2UYOMHXTLRCA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.927467087 (497h7m25.798550122s) [noise=true , address=145.79.227.50:4373 , piece_id=AFFA6ETOV54HUSEAP7J7PAQ3TYUK6NLPMK2V333PKB5MXFTB6RLQ, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.929223062 (497h7m25.800306107s) [noise=true , address=145.79.227.48:4302 , piece_id=VJ7DDH6P5KFTQYYOXIH654Z4CYDR3GFIHPSHALHM7A2XNJMXM3FA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.932350616 (497h7m25.803433661s) [noise=true , address=145.79.227.51:4411 , piece_id=5KFDDQI2CTAJHZA6C3L5TE5YONMMUGHJJS277DEKFHIU6XNOHSSA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.932382367 (497h7m25.803465412s) [noise=true , address=145.79.227.12:4767 , piece_id=R5GHKRYCB4YMZKCOOHNJKXCLWNCUENLS4R5NU5CG7HXDK5NPEG3Q, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.934571415 (497h7m25.80565445s) [noise=true , address=145.79.227.13:4804 , piece_id=NSKUIQ5CU4JACTTEAQOQGMCP35CKWSRQ5LZOXLNZUDDTQU5JFFVA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:25.948870433 (497h7m25.819953468s) [noise=true , address=145.79.227.39:4014 , piece_id=PF64AYWNZQ6L777DMA752M6ICKHZ7Y2CFC27K63FJVJ4JI6VURTA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:26.175501739 (497h7m26.046584784s) [noise=true , address=45.91.116.14:4189 , piece_id=ZMVDHOUM3QGGGOUWNFOM6IGUJJTQ2LIVWTYVT3GKYMYCY3TSKJBQ, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:26.243337510 (497h7m26.114420555s) [noise=true , address=45.91.116.8:4046 , piece_id=QJ55D3XIQARE6UABB2RH4N7M2VRI7OFWI3FHZJNZCHKA2SJMNTCQ, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:26.316426133 (497h7m26.187509178s) [noise=true , address=77.90.31.14:4664 , piece_id=OPFFPWBCSSTJGSK6GUAHZYJ5WAUYVMPSX622VGVDBLKGS2LFAG7Q, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:26.319788446 (497h7m26.190871491s) [noise=true , address=45.91.116.12:4226 , piece_id=RW7MJTDV2PT2JAXIZ3IMN7LXSSSRYOXTJWNKMGLQWGC4K42KUGTA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:26.325360195 (497h7m26.19644323s) [noise=true , address=45.91.116.10:4084 , piece_id=ZD4B23PRY4MXW4ZMSTNZEQ6SXHU4RYKQ7ZXGTTNGO5EAUGEWSCKQ, seg_pos={0 0}, upload=436306]: Done uploading piece. err: <nil>
Dec 26 08:36:26.325377078 (497h7m26.196460113s) [seg_pos={0 0}, upload=436306]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:36:26.325575104 (497h7m26.19665814s) [noise=true , address=77.90.31.17:4773 , piece_id=ZJR3RHK7FJUTHA4HTRTVVAFAAK6GCR3KPU5SI35QGIWRSMHNMOWA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:52544->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:52544->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52544->77.90.31.17:4773: use of closed network connection
Dec 26 08:36:26.325583851 (497h7m26.196666896s) [noise=true , address=77.90.31.17:4773 , piece_id=ZJR3RHK7FJUTHA4HTRTVVAFAAK6GCR3KPU5SI35QGIWRSMHNMOWA, seg_pos={0 0}, upload=436306]: Long tail cancellation due to optimal threshold
Dec 26 08:36:26.325758063 (497h7m26.196841118s) [noise=true , address=77.90.31.10:4519 , piece_id=53F4BGZNBVXO3522TDUDBYAUY36YTDPD6GNDQEHMNS6SJS3Z3GHQ, seg_pos={0 0}, upload=436306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF
Dec 26 08:36:26.325765317 (497h7m26.196848362s) [noise=true , address=77.90.31.10:4519 , piece_id=53F4BGZNBVXO3522TDUDBYAUY36YTDPD6GNDQEHMNS6SJS3Z3GHQ, seg_pos={0 0}, upload=436306]: Long tail cancellation due to optimal threshold
Dec 26 08:36:26.326830585 (497h7m26.19791362s) [noise=true , address=77.90.31.13:4629 , piece_id=2M2RHAGORCTRJ2IBMANNXFE4SCYRCAM6JECRPYMDA5Q2DK3LWKMA, seg_pos={0 0}, upload=436306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:49616->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:49616->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49616->77.90.31.13:4629: use of closed network connection
Dec 26 08:36:26.326853028 (497h7m26.197936063s) [noise=true , address=77.90.31.13:4629 , piece_id=2M2RHAGORCTRJ2IBMANNXFE4SCYRCAM6JECRPYMDA5Q2DK3LWKMA, seg_pos={0 0}, upload=436306]: Long tail cancellation due to optimal threshold
Dec 26 08:36:26.326893745 (497h7m26.197976791s) [seg_pos={0 0}, upload=436306]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:36:26.327030205 (497h7m26.19811324s) [upload=436306]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:36:26.327036116 (497h7m26.198119152s) [upload=436306]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:36:26.619238601 (497h7m26.490321636s) [upload=436306]: Done uploading segments...
Dec 26 08:36:26.619257448 (497h7m26.490340483s) [upload=436306]: Upload finished. err: <nil>
Duration: 30.00495706s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
45.815374317s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:36:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 45.815403131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.523073136s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:36:25 UTC)
orbiter warning: context deadline exceeded
Duration: 5.523157125s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
3.775015298s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:36:12 UTC)
orbiter warning: context deadline exceeded
Duration: 3.77504238s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
45.792311994s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:36:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 45.792328585s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.963122215s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:35:44 UTC)
orbiter warning: context deadline exceeded
Duration: 5.963144818s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...................X..X.XX.X.X..X..X.XXX.....X......X.X...X.] duration:
5.738824763s trace:
https://jaeger.dev.storj.io/trace/4ef3dd01b4780bd6
FAIL (2025-12-26 08:35:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:34:52.028474183 (497h5m51.899557228s) [upload=436285]: Starting upload
Dec 26 08:34:52.028571056 (497h5m51.899654091s) [upload=436285]: Done starting upload
Dec 26 08:34:52.028703218 (497h5m51.899786263s) [upload=436285]: Uploading segments...
Dec 26 08:34:52.028805283 (497h5m51.899888318s) [upload=436285]: Got next segment. Inline: false
Dec 26 08:34:52.028841762 (497h5m51.899924807s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:34:52.028848064 (497h5m51.899931109s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:34:52.272832173 (497h5m52.143915218s) [seg_pos={0 0}, upload=436285]: Begin upload segment...
Dec 26 08:34:52.272888521 (497h5m52.143971556s) [seg_pos={0 0}, upload=436285]: Done begin upload segment.
Dec 26 08:34:52.272951260 (497h5m52.144034295s) [noise=true , address=45.91.116.10:4083 , piece_id=TYXV7LI4TZZ3B3YTGX4MJGAN5BB24BOQXQCKIUD6NPFATLGJU3ZQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.272960347 (497h5m52.144043382s) [noise=true , address=145.79.227.6:4259 , piece_id=AV46K53LGR6XXPBPCIEB7ESIWN3WSGOHC3KTLVXV5GIDDX6GZTZQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.273559718 (497h5m52.144642753s) [noise=true , address=145.79.227.17:4970 , piece_id=6SN355WFP4IA6OCILQJQDKHWC3S3FVEOHMU3KDH5B4KGURYW3YWQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.274022619 (497h5m52.145105665s) [noise=true , address=145.79.227.48:4301 , piece_id=HHERWZPQZ4DVNNSM2DBR5GNEYFT3HQX6TNS4EZ4QFXCTD6JHZTTQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.274558731 (497h5m52.145641766s) [noise=true , address=145.79.227.35:4481 , piece_id=CLMZEZPWF4AG45QPQNN54EIDVXQFKSGJ4BAYFYWUNQGROIVFGLWQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.275034336 (497h5m52.146117401s) [noise=true , address=145.79.227.16:4910 , piece_id=6SHJWLNUZZI7L2WAJAEQTUEWY262QIYYPDVPBYNPRZD3IFY2KOHA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.275484273 (497h5m52.146567308s) [noise=true , address=77.90.31.19:4841 , piece_id=XSNN4PCI7H5H6PB7HMUESXGATQK3QRKMHXQPVCHOLXZBFXRCLYEA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.276603895 (497h5m52.14768693s) [noise=true , address=145.79.227.45:4230 , piece_id=7TJOY63RMWU6NZBGDEKVNPLDI34FGGX65OSSBGIN5IPQNUQDWBFQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.276885041 (497h5m52.147968076s) [noise=true , address=145.79.227.29:4943 , piece_id=XTCB6YWEQ76ICHJVA3DRHSLJGI6F7HGGHPH7AHQ2GOLASEV5WOHQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.277869736 (497h5m52.148952771s) [noise=true , address=145.79.227.36:4519 , piece_id=DUSN5YFRWAOHOTFSLJXE7GL7NOKXVXQUBF54WJV3SOTAV7CQMHXA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.278455040 (497h5m52.149538075s) [noise=true , address=45.91.116.6:4973 , piece_id=WFQ6XRRZ6TQGDWSE2FIQVWM2XXGNLWFKOUY3ULO7BFWYTZVTXQKA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.279167797 (497h5m52.150250832s) [noise=true , address=145.79.227.32:4835 , piece_id=ZXMW3MFRGXOULVJWQMXPDCGZNTRBRG5DN32YUCQC2VP5RENWQSTA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.280267291 (497h5m52.151350326s) [noise=true , address=145.79.227.12:4767 , piece_id=HSHYIVWU4QBRGJLUCUVGY7GMVYXFOTYMDT5GQVPQBRZSSY4CH7LQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.280625573 (497h5m52.151708608s) [noise=true , address=45.91.116.4:4939 , piece_id=J5KQMB3UFOBLUMVJMWSZPNOPPYCZF45227OSMKFT6S7AEC2YFZTA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.281705970 (497h5m52.152789005s) [noise=true , address=145.79.227.39:4013 , piece_id=QJFHKQDY7GHUSSHD4YNRPSEMBILSNGFYCMXWBSFBI3O67M2AKRKQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.282553704 (497h5m52.153636739s) [noise=true , address=77.90.31.16:4736 , piece_id=NNZYKTBGLCKW2F5I2IEHN4URB2NGAWZDA52DJI5PTBH6MAEDB77Q, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:53.389457163 (497h5m53.260540198s) [noise=true , address=77.90.31.19:4841 , piece_id=XSNN4PCI7H5H6PB7HMUESXGATQK3QRKMHXQPVCHOLXZBFXRCLYEA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 26 08:34:54.001087535 (497h5m53.8721706s) [upload=436285]: Next returned nil segment
Dec 26 08:34:54.001102113 (497h5m53.872185158s) [upload=436285]: Waiting for error group managing segments...
Dec 26 08:34:54.282908291 (497h5m54.153991336s) [noise=true , address=145.79.227.29:4943 , piece_id=XTCB6YWEQ76ICHJVA3DRHSLJGI6F7HGGHPH7AHQ2GOLASEV5WOHQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.287076859 (497h5m54.158159904s) [noise=true , address=145.79.227.32:4835 , piece_id=ZXMW3MFRGXOULVJWQMXPDCGZNTRBRG5DN32YUCQC2VP5RENWQSTA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.288489168 (497h5m54.159572203s) [noise=true , address=145.79.227.17:4970 , piece_id=6SN355WFP4IA6OCILQJQDKHWC3S3FVEOHMU3KDH5B4KGURYW3YWQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.290009202 (497h5m54.161092267s) [noise=true , address=145.79.227.35:4481 , piece_id=CLMZEZPWF4AG45QPQNN54EIDVXQFKSGJ4BAYFYWUNQGROIVFGLWQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.297923313 (497h5m54.169006348s) [noise=true , address=145.79.227.39:4013 , piece_id=QJFHKQDY7GHUSSHD4YNRPSEMBILSNGFYCMXWBSFBI3O67M2AKRKQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.298998320 (497h5m54.170081365s) [noise=true , address=145.79.227.36:4519 , piece_id=DUSN5YFRWAOHOTFSLJXE7GL7NOKXVXQUBF54WJV3SOTAV7CQMHXA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.306926527 (497h5m54.178009572s) [noise=true , address=145.79.227.45:4230 , piece_id=7TJOY63RMWU6NZBGDEKVNPLDI34FGGX65OSSBGIN5IPQNUQDWBFQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.308355217 (497h5m54.179438262s) [noise=true , address=145.79.227.6:4259 , piece_id=AV46K53LGR6XXPBPCIEB7ESIWN3WSGOHC3KTLVXV5GIDDX6GZTZQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.311987573 (497h5m54.183070608s) [noise=true , address=145.79.227.48:4301 , piece_id=HHERWZPQZ4DVNNSM2DBR5GNEYFT3HQX6TNS4EZ4QFXCTD6JHZTTQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.324901935 (497h5m54.19598497s) [noise=true , address=145.79.227.12:4767 , piece_id=HSHYIVWU4QBRGJLUCUVGY7GMVYXFOTYMDT5GQVPQBRZSSY4CH7LQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.334696275 (497h5m54.20577932s) [noise=true , address=145.79.227.16:4910 , piece_id=6SHJWLNUZZI7L2WAJAEQTUEWY262QIYYPDVPBYNPRZD3IFY2KOHA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.486548869 (497h5m54.357631914s) [noise=true , address=77.90.31.16:4736 , piece_id=NNZYKTBGLCKW2F5I2IEHN4URB2NGAWZDA52DJI5PTBH6MAEDB77Q, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.528415350 (497h5m54.399498395s) [noise=true , address=45.91.116.6:4973 , piece_id=WFQ6XRRZ6TQGDWSE2FIQVWM2XXGNLWFKOUY3ULO7BFWYTZVTXQKA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.528440919 (497h5m54.399523964s) [seg_pos={0 0}, upload=436285]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:34:54.528735820 (497h5m54.399818855s) [noise=true , address=45.91.116.10:4083 , piece_id=TYXV7LI4TZZ3B3YTGX4MJGAN5BB24BOQXQCKIUD6NPFATLGJU3ZQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 26 08:34:54.528747553 (497h5m54.399830608s) [noise=true , address=45.91.116.10:4083 , piece_id=TYXV7LI4TZZ3B3YTGX4MJGAN5BB24BOQXQCKIUD6NPFATLGJU3ZQ, seg_pos={0 0}, upload=436285]: Long tail cancellation due to optimal threshold
Dec 26 08:34:54.529087970 (497h5m54.400171016s) [noise=true , address=45.91.116.4:4939 , piece_id=J5KQMB3UFOBLUMVJMWSZPNOPPYCZF45227OSMKFT6S7AEC2YFZTA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 26 08:34:54.529109111 (497h5m54.400192176s) [noise=true , address=45.91.116.4:4939 , piece_id=J5KQMB3UFOBLUMVJMWSZPNOPPYCZF45227OSMKFT6S7AEC2YFZTA, seg_pos={0 0}, upload=436285]: Long tail cancellation due to optimal threshold
Dec 26 08:34:54.529177712 (497h5m54.400260757s) [seg_pos={0 0}, upload=436285]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:34:54.529309763 (497h5m54.400392819s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:34:54.529317598 (497h5m54.400400643s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:34:54.833905147 (497h5m54.704988182s) [upload=436285]: Done uploading segments...
Dec 26 08:34:54.833922249 (497h5m54.705005294s) [upload=436285]: Upload finished. err: <nil>
Duration: 5.738841866s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [....................X..X.XX.X.X..X..X.XXX.....X......X.X...X] duration:
30.00438386s trace:
https://jaeger.dev.storj.io/trace/4ef3dd01b4780bd6
FAIL (2025-12-26 08:35:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:34:52.028474183 (497h5m51.899557228s) [upload=436285]: Starting upload
Dec 26 08:34:52.028571056 (497h5m51.899654091s) [upload=436285]: Done starting upload
Dec 26 08:34:52.028703218 (497h5m51.899786263s) [upload=436285]: Uploading segments...
Dec 26 08:34:52.028805283 (497h5m51.899888318s) [upload=436285]: Got next segment. Inline: false
Dec 26 08:34:52.028841762 (497h5m51.899924807s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:34:52.028848064 (497h5m51.899931109s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:34:52.272832173 (497h5m52.143915218s) [seg_pos={0 0}, upload=436285]: Begin upload segment...
Dec 26 08:34:52.272888521 (497h5m52.143971556s) [seg_pos={0 0}, upload=436285]: Done begin upload segment.
Dec 26 08:34:52.272951260 (497h5m52.144034295s) [noise=true , address=45.91.116.10:4083 , piece_id=TYXV7LI4TZZ3B3YTGX4MJGAN5BB24BOQXQCKIUD6NPFATLGJU3ZQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.272960347 (497h5m52.144043382s) [noise=true , address=145.79.227.6:4259 , piece_id=AV46K53LGR6XXPBPCIEB7ESIWN3WSGOHC3KTLVXV5GIDDX6GZTZQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.273559718 (497h5m52.144642753s) [noise=true , address=145.79.227.17:4970 , piece_id=6SN355WFP4IA6OCILQJQDKHWC3S3FVEOHMU3KDH5B4KGURYW3YWQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.274022619 (497h5m52.145105665s) [noise=true , address=145.79.227.48:4301 , piece_id=HHERWZPQZ4DVNNSM2DBR5GNEYFT3HQX6TNS4EZ4QFXCTD6JHZTTQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.274558731 (497h5m52.145641766s) [noise=true , address=145.79.227.35:4481 , piece_id=CLMZEZPWF4AG45QPQNN54EIDVXQFKSGJ4BAYFYWUNQGROIVFGLWQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.275034336 (497h5m52.146117401s) [noise=true , address=145.79.227.16:4910 , piece_id=6SHJWLNUZZI7L2WAJAEQTUEWY262QIYYPDVPBYNPRZD3IFY2KOHA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.275484273 (497h5m52.146567308s) [noise=true , address=77.90.31.19:4841 , piece_id=XSNN4PCI7H5H6PB7HMUESXGATQK3QRKMHXQPVCHOLXZBFXRCLYEA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.276603895 (497h5m52.14768693s) [noise=true , address=145.79.227.45:4230 , piece_id=7TJOY63RMWU6NZBGDEKVNPLDI34FGGX65OSSBGIN5IPQNUQDWBFQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.276885041 (497h5m52.147968076s) [noise=true , address=145.79.227.29:4943 , piece_id=XTCB6YWEQ76ICHJVA3DRHSLJGI6F7HGGHPH7AHQ2GOLASEV5WOHQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.277869736 (497h5m52.148952771s) [noise=true , address=145.79.227.36:4519 , piece_id=DUSN5YFRWAOHOTFSLJXE7GL7NOKXVXQUBF54WJV3SOTAV7CQMHXA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.278455040 (497h5m52.149538075s) [noise=true , address=45.91.116.6:4973 , piece_id=WFQ6XRRZ6TQGDWSE2FIQVWM2XXGNLWFKOUY3ULO7BFWYTZVTXQKA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.279167797 (497h5m52.150250832s) [noise=true , address=145.79.227.32:4835 , piece_id=ZXMW3MFRGXOULVJWQMXPDCGZNTRBRG5DN32YUCQC2VP5RENWQSTA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.280267291 (497h5m52.151350326s) [noise=true , address=145.79.227.12:4767 , piece_id=HSHYIVWU4QBRGJLUCUVGY7GMVYXFOTYMDT5GQVPQBRZSSY4CH7LQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.280625573 (497h5m52.151708608s) [noise=true , address=45.91.116.4:4939 , piece_id=J5KQMB3UFOBLUMVJMWSZPNOPPYCZF45227OSMKFT6S7AEC2YFZTA, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.281705970 (497h5m52.152789005s) [noise=true , address=145.79.227.39:4013 , piece_id=QJFHKQDY7GHUSSHD4YNRPSEMBILSNGFYCMXWBSFBI3O67M2AKRKQ, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:52.282553704 (497h5m52.153636739s) [noise=true , address=77.90.31.16:4736 , piece_id=NNZYKTBGLCKW2F5I2IEHN4URB2NGAWZDA52DJI5PTBH6MAEDB77Q, seg_pos={0 0}, upload=436285]: Uploading piece...
Dec 26 08:34:53.389457163 (497h5m53.260540198s) [noise=true , address=77.90.31.19:4841 , piece_id=XSNN4PCI7H5H6PB7HMUESXGATQK3QRKMHXQPVCHOLXZBFXRCLYEA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 26 08:34:54.001087535 (497h5m53.8721706s) [upload=436285]: Next returned nil segment
Dec 26 08:34:54.001102113 (497h5m53.872185158s) [upload=436285]: Waiting for error group managing segments...
Dec 26 08:34:54.282908291 (497h5m54.153991336s) [noise=true , address=145.79.227.29:4943 , piece_id=XTCB6YWEQ76ICHJVA3DRHSLJGI6F7HGGHPH7AHQ2GOLASEV5WOHQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.287076859 (497h5m54.158159904s) [noise=true , address=145.79.227.32:4835 , piece_id=ZXMW3MFRGXOULVJWQMXPDCGZNTRBRG5DN32YUCQC2VP5RENWQSTA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.288489168 (497h5m54.159572203s) [noise=true , address=145.79.227.17:4970 , piece_id=6SN355WFP4IA6OCILQJQDKHWC3S3FVEOHMU3KDH5B4KGURYW3YWQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.290009202 (497h5m54.161092267s) [noise=true , address=145.79.227.35:4481 , piece_id=CLMZEZPWF4AG45QPQNN54EIDVXQFKSGJ4BAYFYWUNQGROIVFGLWQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.297923313 (497h5m54.169006348s) [noise=true , address=145.79.227.39:4013 , piece_id=QJFHKQDY7GHUSSHD4YNRPSEMBILSNGFYCMXWBSFBI3O67M2AKRKQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.298998320 (497h5m54.170081365s) [noise=true , address=145.79.227.36:4519 , piece_id=DUSN5YFRWAOHOTFSLJXE7GL7NOKXVXQUBF54WJV3SOTAV7CQMHXA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.306926527 (497h5m54.178009572s) [noise=true , address=145.79.227.45:4230 , piece_id=7TJOY63RMWU6NZBGDEKVNPLDI34FGGX65OSSBGIN5IPQNUQDWBFQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.308355217 (497h5m54.179438262s) [noise=true , address=145.79.227.6:4259 , piece_id=AV46K53LGR6XXPBPCIEB7ESIWN3WSGOHC3KTLVXV5GIDDX6GZTZQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.311987573 (497h5m54.183070608s) [noise=true , address=145.79.227.48:4301 , piece_id=HHERWZPQZ4DVNNSM2DBR5GNEYFT3HQX6TNS4EZ4QFXCTD6JHZTTQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.324901935 (497h5m54.19598497s) [noise=true , address=145.79.227.12:4767 , piece_id=HSHYIVWU4QBRGJLUCUVGY7GMVYXFOTYMDT5GQVPQBRZSSY4CH7LQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.334696275 (497h5m54.20577932s) [noise=true , address=145.79.227.16:4910 , piece_id=6SHJWLNUZZI7L2WAJAEQTUEWY262QIYYPDVPBYNPRZD3IFY2KOHA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.486548869 (497h5m54.357631914s) [noise=true , address=77.90.31.16:4736 , piece_id=NNZYKTBGLCKW2F5I2IEHN4URB2NGAWZDA52DJI5PTBH6MAEDB77Q, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.528415350 (497h5m54.399498395s) [noise=true , address=45.91.116.6:4973 , piece_id=WFQ6XRRZ6TQGDWSE2FIQVWM2XXGNLWFKOUY3ULO7BFWYTZVTXQKA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: <nil>
Dec 26 08:34:54.528440919 (497h5m54.399523964s) [seg_pos={0 0}, upload=436285]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:34:54.528735820 (497h5m54.399818855s) [noise=true , address=45.91.116.10:4083 , piece_id=TYXV7LI4TZZ3B3YTGX4MJGAN5BB24BOQXQCKIUD6NPFATLGJU3ZQ, seg_pos={0 0}, upload=436285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 26 08:34:54.528747553 (497h5m54.399830608s) [noise=true , address=45.91.116.10:4083 , piece_id=TYXV7LI4TZZ3B3YTGX4MJGAN5BB24BOQXQCKIUD6NPFATLGJU3ZQ, seg_pos={0 0}, upload=436285]: Long tail cancellation due to optimal threshold
Dec 26 08:34:54.529087970 (497h5m54.400171016s) [noise=true , address=45.91.116.4:4939 , piece_id=J5KQMB3UFOBLUMVJMWSZPNOPPYCZF45227OSMKFT6S7AEC2YFZTA, seg_pos={0 0}, upload=436285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 26 08:34:54.529109111 (497h5m54.400192176s) [noise=true , address=45.91.116.4:4939 , piece_id=J5KQMB3UFOBLUMVJMWSZPNOPPYCZF45227OSMKFT6S7AEC2YFZTA, seg_pos={0 0}, upload=436285]: Long tail cancellation due to optimal threshold
Dec 26 08:34:54.529177712 (497h5m54.400260757s) [seg_pos={0 0}, upload=436285]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:34:54.529309763 (497h5m54.400392819s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:34:54.529317598 (497h5m54.400400643s) [upload=436285]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:34:54.833905147 (497h5m54.704988182s) [upload=436285]: Done uploading segments...
Dec 26 08:34:54.833922249 (497h5m54.705005294s) [upload=436285]: Upload finished. err: <nil>
Duration: 30.004424386s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
48.539648017s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:35:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 48.539672783s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.976210015s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:35:08 UTC)
orbiter warning: context deadline exceeded
Duration: 5.976753361s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
3.474932645s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:34:59 UTC)
orbiter warning: context deadline exceeded
Duration: 3.474960758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.708597521s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:34:43 UTC)
orbiter warning: context deadline exceeded
Duration: 6.709352658s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
49.630635479s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:34:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 49.630668231s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....................X..X.XX.X.X..X..X.XXX.....X......X.X...] duration:
5.727808988s trace:
https://jaeger.dev.storj.io/trace/5b47dd1e6e74f587
FAIL (2025-12-26 08:34:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: FKITZPYN23KG5F746IIFMIX2XP4CNON4CJ4GWHKD7T6YAIGCYNKA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: KSFMLZUV5F2OROGYYLQ2NMHDI7L4Q462FOFKOS3RJSA5EOPYKBNQ) context deadline exceeded
======LOGS======
688816s) [upload=436240]: Starting upload
Dec 26 08:31:32.436662960 (497h2m32.307745995s) [upload=436240]: Done starting upload
Dec 26 08:31:32.436892879 (497h2m32.307975924s) [upload=436240]: Uploading segments...
Dec 26 08:31:32.437119169 (497h2m32.308202204s) [upload=436240]: Got next segment. Inline: false
Dec 26 08:31:32.437146391 (497h2m32.308229436s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:31:32.437150829 (497h2m32.308233864s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:31:32.682925167 (497h2m32.554008202s) [seg_pos={0 0}, upload=436240]: Begin upload segment...
Dec 26 08:31:32.682980282 (497h2m32.554063327s) [seg_pos={0 0}, upload=436240]: Done begin upload segment.
Dec 26 08:31:32.683012464 (497h2m32.554095499s) [noise=true , address=145.79.227.4:4186 , piece_id=2EQO6S57IAHJ7SG27BEFKWGMRXEXHXKPQMGWQ6MOVEJFOPMEIAHQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.683844388 (497h2m32.554927433s) [noise=true , address=145.79.227.35:4482 , piece_id=PSYDVI7KVA2GH5KJJ63XK3IRPH3ONJWP7KYAN5EONYLWICORZVQQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.683872340 (497h2m32.554955385s) [noise=true , address=45.91.116.12:4225 , piece_id=3VWW2XD7O5HVUHQZLTZDTHGTRKQO5INRMKUHX4FUEPVYLQQPM6SA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684275398 (497h2m32.555358433s) [noise=true , address=45.91.116.9:4261 , piece_id=LAVQOYOISKUVK4A3OLTZESBPGRMYO7HQT56ZE6NVFTX6EVN6JIAQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684346754 (497h2m32.555429789s) [noise=true , address=45.91.116.4:4940 , piece_id=U76ENEJCPF6ROLXTLNP6ZGIBHFEDUIBKKH5KRXZOU4XFT7C4X4ZQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684608994 (497h2m32.555692029s) [noise=true , address=145.79.227.11:4730 , piece_id=HTDCULEMM22ZJOTNKSF7665FTJSI7A2KVATDKK7Y6K3A4QS64U2A, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684712912 (497h2m32.555795957s) [noise=true , address=77.90.31.21:4880 , piece_id=ISBGXOEDEXVK6T4WZOJVGSQSQS6DHRAKX5T7NUZV6QDZVZD3T7GA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685017872 (497h2m32.556100907s) [noise=true , address=45.91.116.7:4011 , piece_id=FKITZPYN23KG5F746IIFMIX2XP4CNON4CJ4GWHKD7T6YAIGCYNKA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685085420 (497h2m32.556168455s) [noise=true , address=145.79.227.41:4085 , piece_id=RHB7FBJ72RHH35Q5CMGNEOVQJBRW4Y6TNOH4YM57DOXCLB2DYBUA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685433022 (497h2m32.556516057s) [noise=true , address=45.91.116.11:4120 , piece_id=KSFMLZUV5F2OROGYYLQ2NMHDI7L4Q462FOFKOS3RJSA5EOPYKBNQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685447160 (497h2m32.556530195s) [noise=true , address=45.91.116.8:4047 , piece_id=NW6EQWR5R5VJ7FOMGZENMMQNV6BACE7WSOAGRYTKAWBTXLU3MAXA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685849225 (497h2m32.55693226s) [noise=true , address=145.79.227.15:4874 , piece_id=BP3XSQU2ZTAKZJF3O2X2VVJPMLFOK7SYFO67LRXXBL5PKE34HXRQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685854775 (497h2m32.55693781s) [noise=true , address=45.91.116.14:4191 , piece_id=X6URKIVFRQLNSQBVZETAOO2NX6BXOXBTS6RH7UQH5MUT5JKJEPTQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.686271779 (497h2m32.557354824s) [noise=true , address=77.90.31.19:4841 , piece_id=BIRKVSGCVUEPCWI6Z5PKZNNST76L7AEU27PSYYVWOA6L5GHGUT2A, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.686662574 (497h2m32.557745619s) [noise=true , address=145.79.227.49:4337 , piece_id=Q6PECLDUY6T6O52YN4X6KCXHCUWUIB37MUREWGDWTNKJHQNGXM5A, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.687848382 (497h2m32.558931417s) [noise=true , address=145.79.227.5:4222 , piece_id=O7USVLF657O76N2JCC4NN53AOMNDUDN5IENHLT4ZHLAQ74SJMSJQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:34.393525742 (497h2m34.264608787s) [upload=436240]: Next returned nil segment
Dec 26 08:31:34.393542152 (497h2m34.264625197s) [upload=436240]: Waiting for error group managing segments...
Dec 26 08:31:34.677303745 (497h2m34.54838678s) [noise=true , address=145.79.227.5:4222 , piece_id=O7USVLF657O76N2JCC4NN53AOMNDUDN5IENHLT4ZHLAQ74SJMSJQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.685378611 (497h2m34.556461656s) [noise=true , address=145.79.227.41:4085 , piece_id=RHB7FBJ72RHH35Q5CMGNEOVQJBRW4Y6TNOH4YM57DOXCLB2DYBUA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.687037600 (497h2m34.558120635s) [noise=true , address=145.79.227.11:4730 , piece_id=HTDCULEMM22ZJOTNKSF7665FTJSI7A2KVATDKK7Y6K3A4QS64U2A, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.695849379 (497h2m34.566932424s) [noise=true , address=145.79.227.4:4186 , piece_id=2EQO6S57IAHJ7SG27BEFKWGMRXEXHXKPQMGWQ6MOVEJFOPMEIAHQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.702211644 (497h2m34.573294689s) [noise=true , address=145.79.227.49:4337 , piece_id=Q6PECLDUY6T6O52YN4X6KCXHCUWUIB37MUREWGDWTNKJHQNGXM5A, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.706506232 (497h2m34.577589277s) [noise=true , address=145.79.227.15:4874 , piece_id=BP3XSQU2ZTAKZJF3O2X2VVJPMLFOK7SYFO67LRXXBL5PKE34HXRQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.715758930 (497h2m34.586841966s) [noise=true , address=145.79.227.35:4482 , piece_id=PSYDVI7KVA2GH5KJJ63XK3IRPH3ONJWP7KYAN5EONYLWICORZVQQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.926220557 (497h2m34.797303592s) [noise=true , address=45.91.116.11:4120 , piece_id=KSFMLZUV5F2OROGYYLQ2NMHDI7L4Q462FOFKOS3RJSA5EOPYKBNQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.934954146 (497h2m34.806037191s) [noise=true , address=45.91.116.4:4940 , piece_id=U76ENEJCPF6ROLXTLNP6ZGIBHFEDUIBKKH5KRXZOU4XFT7C4X4ZQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.968439719 (497h2m34.839522774s) [noise=true , address=77.90.31.21:4880 , piece_id=ISBGXOEDEXVK6T4WZOJVGSQSQS6DHRAKX5T7NUZV6QDZVZD3T7GA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.973485226 (497h2m34.844568271s) [noise=true , address=45.91.116.8:4047 , piece_id=NW6EQWR5R5VJ7FOMGZENMMQNV6BACE7WSOAGRYTKAWBTXLU3MAXA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:35.028836676 (497h2m34.899919711s) [noise=true , address=45.91.116.7:4011 , piece_id=FKITZPYN23KG5F746IIFMIX2XP4CNON4CJ4GWHKD7T6YAIGCYNKA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:35.075500752 (497h2m34.946583787s) [noise=true , address=45.91.116.12:4225 , piece_id=3VWW2XD7O5HVUHQZLTZDTHGTRKQO5INRMKUHX4FUEPVYLQQPM6SA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:35.075524267 (497h2m34.946607312s) [seg_pos={0 0}, upload=436240]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:31:35.075760958 (497h2m34.946844003s) [noise=true , address=45.91.116.14:4191 , piece_id=X6URKIVFRQLNSQBVZETAOO2NX6BXOXBTS6RH7UQH5MUT5JKJEPTQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:31:35.075768883 (497h2m34.946851928s) [noise=true , address=45.91.116.14:4191 , piece_id=X6URKIVFRQLNSQBVZETAOO2NX6BXOXBTS6RH7UQH5MUT5JKJEPTQ, seg_pos={0 0}, upload=436240]: Long tail cancellation due to optimal threshold
Dec 26 08:31:35.076085606 (497h2m34.947168661s) [noise=true , address=45.91.116.9:4261 , piece_id=LAVQOYOISKUVK4A3OLTZESBPGRMYO7HQT56ZE6NVFTX6EVN6JIAQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:36826->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:36826->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36826->45.91.116.9:4261: use of closed network connection
Dec 26 08:31:35.076094823 (497h2m34.947177888s) [noise=true , address=45.91.116.9:4261 , piece_id=LAVQOYOISKUVK4A3OLTZESBPGRMYO7HQT56ZE6NVFTX6EVN6JIAQ, seg_pos={0 0}, upload=436240]: Long tail cancellation due to optimal threshold
Dec 26 08:31:35.076197258 (497h2m34.947280303s) [noise=true , address=77.90.31.19:4841 , piece_id=BIRKVSGCVUEPCWI6Z5PKZNNST76L7AEU27PSYYVWOA6L5GHGUT2A, seg_pos={0 0}, upload=436240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:35706->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 08:31:35.076204962 (497h2m34.947288028s) [noise=true , address=77.90.31.19:4841 , piece_id=BIRKVSGCVUEPCWI6Z5PKZNNST76L7AEU27PSYYVWOA6L5GHGUT2A, seg_pos={0 0}, upload=436240]: Long tail cancellation due to optimal threshold
Dec 26 08:31:35.076224220 (497h2m34.947307265s) [seg_pos={0 0}, upload=436240]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:31:35.076310694 (497h2m34.947393749s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:31:35.076317367 (497h2m34.947400422s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:31:35.367929207 (497h2m35.239012242s) [upload=436240]: Done uploading segments...
Dec 26 08:31:35.367950037 (497h2m35.239033072s) [upload=436240]: Upload finished. err: <nil>
Duration: 5.727832933s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................X..................................] duration:
3.475325253s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:34:02 UTC)
orbiter warning: context deadline exceeded
Duration: 3.475357203s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.247452891s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:33:45 UTC)
orbiter warning: context deadline exceeded
Duration: 6.24747903s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.002756644s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:33:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 1m0.002768146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................X.................................] duration:
3.519675012s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:33:30 UTC)
orbiter warning: context deadline exceeded
Duration: 3.519706542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.001880467s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:33:23 UTC)
orbiter warning: context deadline exceeded
Duration: 6.001916577s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [......................X..X.XX.X.X..X..X.XXX.....X......X.X..] duration:
6.001823628s trace:
https://jaeger.dev.storj.io/trace/5b47dd1e6e74f587
FAIL (2025-12-26 08:33:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: FKITZPYN23KG5F746IIFMIX2XP4CNON4CJ4GWHKD7T6YAIGCYNKA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: KSFMLZUV5F2OROGYYLQ2NMHDI7L4Q462FOFKOS3RJSA5EOPYKBNQ) context deadline exceeded
======LOGS======
688816s) [upload=436240]: Starting upload
Dec 26 08:31:32.436662960 (497h2m32.307745995s) [upload=436240]: Done starting upload
Dec 26 08:31:32.436892879 (497h2m32.307975924s) [upload=436240]: Uploading segments...
Dec 26 08:31:32.437119169 (497h2m32.308202204s) [upload=436240]: Got next segment. Inline: false
Dec 26 08:31:32.437146391 (497h2m32.308229436s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:31:32.437150829 (497h2m32.308233864s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:31:32.682925167 (497h2m32.554008202s) [seg_pos={0 0}, upload=436240]: Begin upload segment...
Dec 26 08:31:32.682980282 (497h2m32.554063327s) [seg_pos={0 0}, upload=436240]: Done begin upload segment.
Dec 26 08:31:32.683012464 (497h2m32.554095499s) [noise=true , address=145.79.227.4:4186 , piece_id=2EQO6S57IAHJ7SG27BEFKWGMRXEXHXKPQMGWQ6MOVEJFOPMEIAHQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.683844388 (497h2m32.554927433s) [noise=true , address=145.79.227.35:4482 , piece_id=PSYDVI7KVA2GH5KJJ63XK3IRPH3ONJWP7KYAN5EONYLWICORZVQQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.683872340 (497h2m32.554955385s) [noise=true , address=45.91.116.12:4225 , piece_id=3VWW2XD7O5HVUHQZLTZDTHGTRKQO5INRMKUHX4FUEPVYLQQPM6SA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684275398 (497h2m32.555358433s) [noise=true , address=45.91.116.9:4261 , piece_id=LAVQOYOISKUVK4A3OLTZESBPGRMYO7HQT56ZE6NVFTX6EVN6JIAQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684346754 (497h2m32.555429789s) [noise=true , address=45.91.116.4:4940 , piece_id=U76ENEJCPF6ROLXTLNP6ZGIBHFEDUIBKKH5KRXZOU4XFT7C4X4ZQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684608994 (497h2m32.555692029s) [noise=true , address=145.79.227.11:4730 , piece_id=HTDCULEMM22ZJOTNKSF7665FTJSI7A2KVATDKK7Y6K3A4QS64U2A, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.684712912 (497h2m32.555795957s) [noise=true , address=77.90.31.21:4880 , piece_id=ISBGXOEDEXVK6T4WZOJVGSQSQS6DHRAKX5T7NUZV6QDZVZD3T7GA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685017872 (497h2m32.556100907s) [noise=true , address=45.91.116.7:4011 , piece_id=FKITZPYN23KG5F746IIFMIX2XP4CNON4CJ4GWHKD7T6YAIGCYNKA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685085420 (497h2m32.556168455s) [noise=true , address=145.79.227.41:4085 , piece_id=RHB7FBJ72RHH35Q5CMGNEOVQJBRW4Y6TNOH4YM57DOXCLB2DYBUA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685433022 (497h2m32.556516057s) [noise=true , address=45.91.116.11:4120 , piece_id=KSFMLZUV5F2OROGYYLQ2NMHDI7L4Q462FOFKOS3RJSA5EOPYKBNQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685447160 (497h2m32.556530195s) [noise=true , address=45.91.116.8:4047 , piece_id=NW6EQWR5R5VJ7FOMGZENMMQNV6BACE7WSOAGRYTKAWBTXLU3MAXA, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685849225 (497h2m32.55693226s) [noise=true , address=145.79.227.15:4874 , piece_id=BP3XSQU2ZTAKZJF3O2X2VVJPMLFOK7SYFO67LRXXBL5PKE34HXRQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.685854775 (497h2m32.55693781s) [noise=true , address=45.91.116.14:4191 , piece_id=X6URKIVFRQLNSQBVZETAOO2NX6BXOXBTS6RH7UQH5MUT5JKJEPTQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.686271779 (497h2m32.557354824s) [noise=true , address=77.90.31.19:4841 , piece_id=BIRKVSGCVUEPCWI6Z5PKZNNST76L7AEU27PSYYVWOA6L5GHGUT2A, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.686662574 (497h2m32.557745619s) [noise=true , address=145.79.227.49:4337 , piece_id=Q6PECLDUY6T6O52YN4X6KCXHCUWUIB37MUREWGDWTNKJHQNGXM5A, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:32.687848382 (497h2m32.558931417s) [noise=true , address=145.79.227.5:4222 , piece_id=O7USVLF657O76N2JCC4NN53AOMNDUDN5IENHLT4ZHLAQ74SJMSJQ, seg_pos={0 0}, upload=436240]: Uploading piece...
Dec 26 08:31:34.393525742 (497h2m34.264608787s) [upload=436240]: Next returned nil segment
Dec 26 08:31:34.393542152 (497h2m34.264625197s) [upload=436240]: Waiting for error group managing segments...
Dec 26 08:31:34.677303745 (497h2m34.54838678s) [noise=true , address=145.79.227.5:4222 , piece_id=O7USVLF657O76N2JCC4NN53AOMNDUDN5IENHLT4ZHLAQ74SJMSJQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.685378611 (497h2m34.556461656s) [noise=true , address=145.79.227.41:4085 , piece_id=RHB7FBJ72RHH35Q5CMGNEOVQJBRW4Y6TNOH4YM57DOXCLB2DYBUA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.687037600 (497h2m34.558120635s) [noise=true , address=145.79.227.11:4730 , piece_id=HTDCULEMM22ZJOTNKSF7665FTJSI7A2KVATDKK7Y6K3A4QS64U2A, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.695849379 (497h2m34.566932424s) [noise=true , address=145.79.227.4:4186 , piece_id=2EQO6S57IAHJ7SG27BEFKWGMRXEXHXKPQMGWQ6MOVEJFOPMEIAHQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.702211644 (497h2m34.573294689s) [noise=true , address=145.79.227.49:4337 , piece_id=Q6PECLDUY6T6O52YN4X6KCXHCUWUIB37MUREWGDWTNKJHQNGXM5A, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.706506232 (497h2m34.577589277s) [noise=true , address=145.79.227.15:4874 , piece_id=BP3XSQU2ZTAKZJF3O2X2VVJPMLFOK7SYFO67LRXXBL5PKE34HXRQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.715758930 (497h2m34.586841966s) [noise=true , address=145.79.227.35:4482 , piece_id=PSYDVI7KVA2GH5KJJ63XK3IRPH3ONJWP7KYAN5EONYLWICORZVQQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.926220557 (497h2m34.797303592s) [noise=true , address=45.91.116.11:4120 , piece_id=KSFMLZUV5F2OROGYYLQ2NMHDI7L4Q462FOFKOS3RJSA5EOPYKBNQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.934954146 (497h2m34.806037191s) [noise=true , address=45.91.116.4:4940 , piece_id=U76ENEJCPF6ROLXTLNP6ZGIBHFEDUIBKKH5KRXZOU4XFT7C4X4ZQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.968439719 (497h2m34.839522774s) [noise=true , address=77.90.31.21:4880 , piece_id=ISBGXOEDEXVK6T4WZOJVGSQSQS6DHRAKX5T7NUZV6QDZVZD3T7GA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:34.973485226 (497h2m34.844568271s) [noise=true , address=45.91.116.8:4047 , piece_id=NW6EQWR5R5VJ7FOMGZENMMQNV6BACE7WSOAGRYTKAWBTXLU3MAXA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:35.028836676 (497h2m34.899919711s) [noise=true , address=45.91.116.7:4011 , piece_id=FKITZPYN23KG5F746IIFMIX2XP4CNON4CJ4GWHKD7T6YAIGCYNKA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:35.075500752 (497h2m34.946583787s) [noise=true , address=45.91.116.12:4225 , piece_id=3VWW2XD7O5HVUHQZLTZDTHGTRKQO5INRMKUHX4FUEPVYLQQPM6SA, seg_pos={0 0}, upload=436240]: Done uploading piece. err: <nil>
Dec 26 08:31:35.075524267 (497h2m34.946607312s) [seg_pos={0 0}, upload=436240]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:31:35.075760958 (497h2m34.946844003s) [noise=true , address=45.91.116.14:4191 , piece_id=X6URKIVFRQLNSQBVZETAOO2NX6BXOXBTS6RH7UQH5MUT5JKJEPTQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:31:35.075768883 (497h2m34.946851928s) [noise=true , address=45.91.116.14:4191 , piece_id=X6URKIVFRQLNSQBVZETAOO2NX6BXOXBTS6RH7UQH5MUT5JKJEPTQ, seg_pos={0 0}, upload=436240]: Long tail cancellation due to optimal threshold
Dec 26 08:31:35.076085606 (497h2m34.947168661s) [noise=true , address=45.91.116.9:4261 , piece_id=LAVQOYOISKUVK4A3OLTZESBPGRMYO7HQT56ZE6NVFTX6EVN6JIAQ, seg_pos={0 0}, upload=436240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:36826->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:36826->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36826->45.91.116.9:4261: use of closed network connection
Dec 26 08:31:35.076094823 (497h2m34.947177888s) [noise=true , address=45.91.116.9:4261 , piece_id=LAVQOYOISKUVK4A3OLTZESBPGRMYO7HQT56ZE6NVFTX6EVN6JIAQ, seg_pos={0 0}, upload=436240]: Long tail cancellation due to optimal threshold
Dec 26 08:31:35.076197258 (497h2m34.947280303s) [noise=true , address=77.90.31.19:4841 , piece_id=BIRKVSGCVUEPCWI6Z5PKZNNST76L7AEU27PSYYVWOA6L5GHGUT2A, seg_pos={0 0}, upload=436240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:35706->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 08:31:35.076204962 (497h2m34.947288028s) [noise=true , address=77.90.31.19:4841 , piece_id=BIRKVSGCVUEPCWI6Z5PKZNNST76L7AEU27PSYYVWOA6L5GHGUT2A, seg_pos={0 0}, upload=436240]: Long tail cancellation due to optimal threshold
Dec 26 08:31:35.076224220 (497h2m34.947307265s) [seg_pos={0 0}, upload=436240]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:31:35.076310694 (497h2m34.947393749s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:31:35.076317367 (497h2m34.947400422s) [upload=436240]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:31:35.367929207 (497h2m35.239012242s) [upload=436240]: Done uploading segments...
Dec 26 08:31:35.367950037 (497h2m35.239033072s) [upload=436240]: Upload finished. err: <nil>
Duration: 6.001873684s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.901084485s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:32:29 UTC)
orbiter warning: context deadline exceeded
Duration: 5.901791261s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................X................................] duration:
3.517362758s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:32:22 UTC)
orbiter warning: context deadline exceeded
Duration: 3.517464752s
core
uplink
us-select-1
us1-placement-20
|