Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
30.000458874s
FAIL (2025-09-06 09:27:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000479214s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
2m10.229089839s
FAIL (2025-09-06 09:27:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0)
======LOGS======
2UU2JQ3TRC4H5L35RUICWTXXYYDRXI5MDQA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ, address:145.79.227.34:4918): protocol: write tcp 188.166.225.170:55984->145.79.227.34:4918: use of closed network connection; write tcp 188.166.225.170:55984->145.79.227.34:4918: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55984->145.79.227.34:4918: use of closed network connection
Sep 6 09:25:57.150727104 (187h17m53.830922663s) [noise=true , address=145.79.227.18:4016 , piece_id=XIU6ZSP53NOAZZLRUCX6YXPTDU6OLZZJNFFRTHRJUUHDSENEVKXA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:145.79.227.18:4016): protocol: read tcp 188.166.225.170:53988->145.79.227.18:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.150777580 (187h17m53.830973149s) [noise=true , address=145.79.227.13:4813 , piece_id=4QIR4W4HOSOZGMBIHPCESC4PIUOLDITQLK46PWR3TUO5TTNNVGVA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:145.79.227.13:4813): protocol: write tcp 188.166.225.170:36266->145.79.227.13:4813: use of closed network connection; write tcp 188.166.225.170:36266->145.79.227.13:4813: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36266->145.79.227.13:4813: use of closed network connection
Sep 6 09:25:57.150854838 (187h17m53.831050407s) [noise=true , address=145.79.227.6:4288 , piece_id=LFOQN6BRLN236CAYJTXOXNVCUEV2YVGBNCUYTT2GNHS7VW5B5OWQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:145.79.227.6:4288): protocol: write tcp 188.166.225.170:60002->145.79.227.6:4288: use of closed network connection; write tcp 188.166.225.170:60002->145.79.227.6:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60002->145.79.227.6:4288: use of closed network connection
Sep 6 09:25:57.151007570 (187h17m53.831203139s) [noise=true , address=45.91.116.9:4292 , piece_id=Y3HI7ZZTXH267753D2D6DFY3OPNRWN3JJZ5JNHUQ7ON7XLKPBSKQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12tnWSKAebAwLeGWGRsNnYpcWgbX5UHSVRUnnu9qesUZCK4vY81, address:45.91.116.9:4292): protocol: write tcp 188.166.225.170:51364->45.91.116.9:4292: use of closed network connection; write tcp 188.166.225.170:51364->45.91.116.9:4292: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51364->45.91.116.9:4292: use of closed network connection
Sep 6 09:25:57.151045333 (187h17m53.831240902s) [noise=true , address=145.79.227.14:4851 , piece_id=3QCHX47ZM5755W7FXVQW3VVWBHGPNI64IO5BYMRY4RBBK27CZLRQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:145.79.227.14:4851): protocol: write tcp 188.166.225.170:39650->145.79.227.14:4851: use of closed network connection; write tcp 188.166.225.170:39650->145.79.227.14:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39650->145.79.227.14:4851: use of closed network connection
Sep 6 09:25:57.151055662 (187h17m53.831251231s) [noise=true , address=77.90.31.17:4786 , piece_id=CUDYBW425IEO5ZSLGSJ2LFYKOGG7RVOS22GRK3T5NTDMILTDO7NQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 188.166.225.170:55822->77.90.31.17:4786: use of closed network connection; write tcp 188.166.225.170:55822->77.90.31.17:4786: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55822->77.90.31.17:4786: use of closed network connection
Sep 6 09:25:57.151161966 (187h17m53.831357535s) [noise=true , address=145.79.227.42:4149 , piece_id=VQMXMQQWSVQ5FRWRMOUXQC2CCRWKGKB6RISVRJ4UDAZNLH27MQAA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:145.79.227.42:4149): protocol: write tcp 188.166.225.170:42132->145.79.227.42:4149: use of closed network connection; write tcp 188.166.225.170:42132->145.79.227.42:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42132->145.79.227.42:4149: use of closed network connection
Sep 6 09:25:57.151253420 (187h17m53.83144899s) [noise=true , address=145.79.227.41:4117 , piece_id=Q23E4DIJJXM3OTJZ56GGB5MISTCAAPZKERQOQRWHR56RMYTZBGSA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: read tcp 188.166.225.170:56326->145.79.227.41:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151285042 (187h17m53.831480611s) [noise=true , address=45.91.116.13:4163 , piece_id=V7NUE2V7PBKT2F7TSVPN7GH42HK2O3V7I4MSVSS5BPQ5PC3ZD4VQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1hRpjHEvHzBoYqq6DXKC5Z3SPsP4HEZz2tAiuheHkgzzcnsVU2, address:45.91.116.13:4163): protocol: read tcp 188.166.225.170:45312->45.91.116.13:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151316742 (187h17m53.831512311s) [noise=true , address=145.79.227.33:4873 , piece_id=WX3SZAGVGRKZNG7DOW3WGUBQEHIQMPPB6KVZBRNVV7OL2SSVJ7VQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: read tcp 188.166.225.170:40194->145.79.227.33:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151375825 (187h17m53.831571394s) [noise=true , address=45.91.116.10:4086 , piece_id=2TA5PFCAG4OBXIYQJPOUPDTEASOG7P3WTKM5HOPBFW3HZOWUB7CA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: write tcp 188.166.225.170:49946->45.91.116.10:4086: use of closed network connection; write tcp 188.166.225.170:49946->45.91.116.10:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49946->45.91.116.10:4086: use of closed network connection
Sep 6 09:25:57.151449867 (187h17m53.831645426s) [noise=true , address=145.79.227.5:4247 , piece_id=QQZAPJ6KDR2IH72WSBJANEFN674Y4DLZ4H2DCPF2C246SMYQNL2A, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: write tcp 188.166.225.170:43138->145.79.227.5:4247: use of closed network connection; write tcp 188.166.225.170:43138->145.79.227.5:4247: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43138->145.79.227.5:4247: use of closed network connection
Sep 6 09:25:57.151546111 (187h17m53.83174168s) [noise=true , address=145.79.227.52:4466 , piece_id=T2REC67ZLNH2QNAT34FLTL3TVL5IUWDJEXVIYN4WI6KM22AH7ZEQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP, address:145.79.227.52:4466): protocol: read tcp 188.166.225.170:57676->145.79.227.52:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151572491 (187h17m53.83176806s) [noise=true , address=145.79.227.30:4780 , piece_id=PWNXYAHCQOC2CRPC53NTQRODOQIQZ5WXN5ZZQ75E3HA6OYDJESLQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:36472->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151605845 (187h17m53.831801404s) [noise=true , address=145.79.227.15:4899 , piece_id=DINZEEUNQOHO7B2GJ37G54V4DSFKIICLHMRQABQIB4VFRMI3UCTQ, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:145.79.227.15:4899): protocol: read tcp 188.166.225.170:47364->145.79.227.15:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151613781 (187h17m53.83180934s) [noise=true , address=145.79.227.19:4088 , piece_id=555QHJJ7BEYNZ6N4FZIDIJAPUHLOZLYJ3ZZDCUIZGYULFTWS72OA, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, address:145.79.227.19:4088): protocol: read tcp 188.166.225.170:49024->145.79.227.19:4088: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151652064 (187h17m53.831847633s) [noise=true , address=145.79.227.37:4578 , piece_id=FN63UQCLJ3TTE2IY6X4IHFR3ZWGMNPZK2JFX5DSTVIROIMNONK5Q, seg_pos={0 0}, upload=164838]: Done uploading piece. err: ecclient: upload failed (node:12cpA79fa3pVKrwiXD2vAXJXGiwtkRMzUvmphKN27n5H4YgtUUK, address:145.79.227.37:4578): protocol: read tcp 188.166.225.170:45410->145.79.227.37:4578: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:57.151687341 (187h17m53.83188291s) [seg_pos={0 0}, upload=164838]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil>
Duration: 2m10.229111913s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................X.X] duration:
30.00085785s
FAIL (2025-09-06 09:26:47 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 30.000882937s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001350169s
FAIL (2025-09-06 09:26:45 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:26:15.578455052 (187h18m12.258650621s) [upload=164851]: Starting upload Sep 6 09:26:15.578519815 (187h18m12.258715384s) [upload=164851]: Done starting upload Sep 6 09:26:15.578590832 (187h18m12.258786401s) [upload=164851]: Uploading segments... Sep 6 09:26:15.578676165 (187h18m12.258871734s) [upload=164851]: Got next segment. Inline: true Sep 6 09:26:15.578681505 (187h18m12.258877074s) [upload=164851]: Waiting for error group managing segments... Sep 6 09:26:15.578742011 (187h18m12.25893758s) [upload=164851]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:26:15.578747191 (187h18m12.25894276s) [upload=164851]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:26:15.578750207 (187h18m12.258945776s) [upload=164851]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:26:28.462130697 (187h18m25.142326266s) [upload=164851]: Done uploading segments... Sep 6 09:26:28.462148992 (187h18m25.142344561s) [upload=164851]: Upload finished. err: <nil>
Duration: 1m0.001404914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.175807009s
FAIL (2025-09-06 09:26:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 10)
======LOGS======
iece_id=DPBPKT5WZJHFTD46OCP552YHEBN2ZTLSTQOKMVFXK52534CXAGWA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: <nil>
Sep 6 09:25:46.939837911 (187h17m43.62003348s) [noise=true , address=89.149.201.147:4006 , piece_id=B5WSNCFVPOOMNMVZUA4DX22WBO6AE5YACWLETWM7OCU74CJWBHWA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: <nil>
Sep 6 09:25:46.969841735 (187h17m43.650037304s) [noise=true , address=89.149.201.147:4016 , piece_id=LB42PRL56NGXDV5VWOAWC5IZIM53X6YKWQC6GYVDQWWXITILXQIQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: <nil>
Sep 6 09:26:44.360090975 (187h18m41.040286544s) [noise=true , address=185.167.204.205:4010 , piece_id=KCAZGBPS4RSV3RMYTVXS27XREPIHJTUCNAHUCPA4ECEILIXO2QOQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: read tcp 188.166.225.170:46768->185.167.204.205:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.360392602 (187h18m41.040588171s) [noise=true , address=81.171.14.70:4043 , piece_id=SMY2R2X3BHQWOF6DYG3WSVTKDEWUXVLHOAQYHCQXXEKY7SC5DIWA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: write tcp 188.166.225.170:59744->81.171.14.70:4043: use of closed network connection; write tcp 188.166.225.170:59744->81.171.14.70:4043: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59744->81.171.14.70:4043: use of closed network connection
Sep 6 09:26:44.360715040 (187h18m41.040910609s) [noise=true , address=185.167.204.206:4047 , piece_id=EBXT3KL7NLFYINJQVWI7IXV6OCNE5GHZMA6UGI6CQO4XZ3TT7VAA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12M7W4Yno5rMgemSC7nJd42asmZXqUY9cCyY7mtFFHEh8iqAhCF, address:185.167.204.206:4047): protocol: read tcp 188.166.225.170:45812->185.167.204.206:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.360764344 (187h18m41.040959913s) [noise=true , address=185.167.204.206:4057 , piece_id=Q2LWRED74GMBKDUUIZUISCTEL4YJNF5CCW4H7BVQ2FSK5XCNPZOQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: read tcp 188.166.225.170:60758->185.167.204.206:4057: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.360804961 (187h18m41.04100053s) [noise=true , address=185.167.204.206:4072 , piece_id=UBKATBCYEACDMBG26JD2D2BQDBEJJZUAB3FNSIXMSXHJODKC4QMQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: read tcp 188.166.225.170:41652->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.360842474 (187h18m41.041038033s) [noise=true , address=185.167.204.206:4054 , piece_id=GVQGN2JWUUUUO6XW3BQQGJJPBRXXVGOKCYG7JZ2HUYHGLOMBKXXQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: read tcp 188.166.225.170:54188->185.167.204.206:4054: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.360893531 (187h18m41.0410891s) [noise=true , address=81.171.14.70:4025 , piece_id=ZWGHPD3ZA76D3BRSPEAGSHXMQLCD4BG6WK4ZUXQ5ST7PEJDAWI5A, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, address:81.171.14.70:4025): protocol: read tcp 188.166.225.170:60836->81.171.14.70:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361029180 (187h18m41.041224749s) [noise=true , address=81.171.14.70:4026 , piece_id=LKVY7RWZHMNUS45OEZAB2KAH25YPJNHRNOMMMG56DCODY5U7R2ZA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:56176->81.171.14.70:4026: use of closed network connection; write tcp 188.166.225.170:56176->81.171.14.70:4026: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56176->81.171.14.70:4026: use of closed network connection
Sep 6 09:26:44.361074417 (187h18m41.041269976s) [noise=true , address=185.167.204.205:4007 , piece_id=QYSX5EMJH3BIUKA2FOAHJ3END5TZJB33PDLCGTFYKNIWE5A3G5AQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12m7e6EGQBC1wvbRQ5uwG8wuHbgZeyXPdj5XDMi1kgo5woPgXR, address:185.167.204.205:4007): protocol: read tcp 188.166.225.170:36874->185.167.204.205:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361195970 (187h18m41.041391549s) [noise=true , address=185.167.204.206:4061 , piece_id=DZX54HEVTFOHXYXIYMCEYIVMQHBDWNSGC2QP7KYKGOK6BYFD3THQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1G18tJ6BsYF3A1sLpRTTGzomqRWsRsj8tbHMJQeR1tZunoTDia, address:185.167.204.206:4061): protocol: read tcp 188.166.225.170:60820->185.167.204.206:4061: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361299858 (187h18m41.041495427s) [noise=true , address=81.171.14.70:4037 , piece_id=WBYDJ4H3W3YQ7AL6B5N3JNVMN5UWBHNU2MMUVSFXY4B7SFJYMFQQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx, address:81.171.14.70:4037): protocol: read tcp 188.166.225.170:41752->81.171.14.70:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361372818 (187h18m41.041568387s) [noise=true , address=81.171.14.70:4046 , piece_id=Y5E7XEDPYKRKGARZIWCW67WXQDHCQOVKRMBRYVPKGCYPNLULFUOQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7, address:81.171.14.70:4046): protocol: read tcp 188.166.225.170:52558->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361432913 (187h18m41.041628492s) [noise=true , address=185.167.204.205:4033 , piece_id=FJ5HQZPL23Z6BRKJF4XTYKNF3XOG5S4DGPUSKG5RLUB5XYKFAEIA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: read tcp 188.166.225.170:43454->185.167.204.205:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361489531 (187h18m41.04168512s) [noise=true , address=81.171.14.70:4042 , piece_id=QZG5OBKZACERILXCPZYC2LRYVKYQ3S7JC5TQS356TAUHRV2Y2NDQ, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:39658->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361529438 (187h18m41.041725007s) [noise=true , address=185.167.204.206:4046 , piece_id=IQQZG4BYLO4GHJQEIIBOWHANY4ZYNI4GW62CGIXSCYHPMQFBVJGA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1YPnmxhn6aS6pLJ7VhwQkHrefetEQwH3ww3sBRyEghNvhzT8X5, address:185.167.204.206:4046): protocol: read tcp 188.166.225.170:36564->185.167.204.206:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361622025 (187h18m41.041817604s) [noise=true , address=81.171.14.70:4033 , piece_id=YHWSCOKVQ74AICIJUBR456FYDEI7FEQL3OTVNOITDPHN2IG7OV6Q, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8, address:81.171.14.70:4033): protocol: read tcp 188.166.225.170:47216->81.171.14.70:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361675587 (187h18m41.041871156s) [noise=true , address=185.167.204.205:4017 , piece_id=P6PID7EZ2CCN2FXZUJJBWVLYDMVRF5KP3GGE4AV7KOIXHP5BWZRA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12em75cRFPaX8jTLrVV4sbuL4AgzzHopjFfiJ6MsvvYKKGNFe8h, address:185.167.204.205:4017): protocol: read tcp 188.166.225.170:44796->185.167.204.205:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361708510 (187h18m41.041904079s) [noise=true , address=185.167.204.205:4020 , piece_id=5VXM6DHEAS6LMCJOORVKZXZOIES4BUCRDZUHKIQRZLDFWK3IW3FA, seg_pos={0 0}, upload=164846]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: read tcp 188.166.225.170:37278->185.167.204.205:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:44.361742034 (187h18m41.041937603s) [seg_pos={0 0}, upload=164846]: Done waiting for segment. successful: 10 optimal: 26 errs: <nil>
Sep 6 09:26:44.534558568 (187h18m41.214754137s) [upload=164846]: Done uploading segments...
Sep 6 09:26:44.534583435 (187h18m41.214779005s) [upload=164846]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 10)
Duration: 1m0.175820715s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
30.000658822s
FAIL (2025-09-06 09:26:42 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:26:24.530933069 (187h18m21.211128628s) [upload=164853]: Starting upload Sep 6 09:26:24.530983696 (187h18m21.211179265s) [upload=164853]: Done starting upload Sep 6 09:26:24.531031518 (187h18m21.211227087s) [upload=164853]: Uploading segments... Sep 6 09:26:24.531066474 (187h18m21.211262043s) [upload=164853]: Got next segment. Inline: true Sep 6 09:26:24.531070081 (187h18m21.21126565s) [upload=164853]: Waiting for error group managing segments... Sep 6 09:26:24.531102564 (187h18m21.211298133s) [upload=164853]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:26:24.531106511 (187h18m21.211302081s) [upload=164853]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:26:24.531109036 (187h18m21.211304605s) [upload=164853]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:26:27.905208418 (187h18m24.585403987s) [upload=164853]: Done uploading segments... Sep 6 09:26:27.905229309 (187h18m24.585424878s) [upload=164853]: Upload finished. err: <nil>
Duration: 30.000679471s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
31.390114421s
FAIL (2025-09-06 09:26:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0)
======LOGS======
55437s) [noise=true , address=45.91.116.13:4155 , piece_id=4FXBSWJTEKG77FECZLSSM5IQW6H4CWRRX53FONWRTSPD65YIY5TA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.812870410 (187h18m28.493065979s) [noise=true , address=77.90.31.17:4771 , piece_id=EKO76OTJ3IOQCDF24AQ4RYBGGHY4SCREYKJKGDFRIOQBZVHD3ISA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813015227 (187h18m28.493210796s) [noise=true , address=77.90.31.14:4661 , piece_id=H33CIU52RM7ULRVA6TPCGUBNZO7WRZU3X2FBLG5U42MXDKKVEBLA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813136027 (187h18m28.493331596s) [noise=true , address=45.91.116.12:4227 , piece_id=JERDILAP2VBG7BBKCZFVBPA4FAAITBMRVPN3KQ27RG245ZL54C6A, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813312145 (187h18m28.493507704s) [noise=true , address=45.91.116.11:4119 , piece_id=O2ZRRTZA3PE2JE5CZIURS6YB2UE6BRLZYSHHWI25GQUXQ6M42HWA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813399512 (187h18m28.493595081s) [noise=true , address=45.91.116.14:4191 , piece_id=NC7UWT6MQJX5FEVZWZ7TUH3VET3HRD3W3GKXLPAFF2YLPKD4E2FA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813565339 (187h18m28.493760908s) [noise=true , address=77.90.31.14:4662 , piece_id=4QDVIDURB7AOZLX6F6J7YHXTSOIPYUYKRTUSV62XCSXGNWC5YBNA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813696681 (187h18m28.49389225s) [noise=true , address=45.91.116.12:4226 , piece_id=MCRMWZYLQHMQY2GKXQRL3L7I4CZCF2TXTQ2QQWDYTFYHDLUV66KA, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.813769641 (187h18m28.49396521s) [noise=true , address=77.90.31.12:4590 , piece_id=7OVJDO6JKOOPYPIZJZGAD4XFWTRWS6IPQ66UKQYEUS5DWGCMHXZQ, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:31.814011372 (187h18m28.494206941s) [noise=true , address=45.91.116.8:4047 , piece_id=SV2WQI7MVKWBPHQ3QME6HL3JGRIEIKDEBCOJN7BBC7VOUF2NIH4Q, seg_pos={0 0}, upload=164850]: Uploading piece...
Sep 6 09:26:35.761638148 (187h18m32.441833717s) [noise=true , address=45.91.116.11:4117 , piece_id=RH2D2U762O6RAR4XLO5GH7MCU6GYKW2GG265P4FLRIC2VR6LJ6KA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 188.166.225.170:42146->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761695237 (187h18m32.441890806s) [noise=true , address=45.91.116.11:4118 , piece_id=KOMCLRKOCECQABBOG2RW2VC4J7IQ2YWPVM3ML6TOD2C3YOPTMTAA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:55248->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761738138 (187h18m32.441933708s) [noise=true , address=45.91.116.13:4155 , piece_id=4FXBSWJTEKG77FECZLSSM5IQW6H4CWRRX53FONWRTSPD65YIY5TA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 188.166.225.170:38512->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761767284 (187h18m32.441962843s) [noise=true , address=45.91.116.12:4226 , piece_id=MCRMWZYLQHMQY2GKXQRL3L7I4CZCF2TXTQ2QQWDYTFYHDLUV66KA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:47792->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761827559 (187h18m32.442023128s) [noise=true , address=77.90.31.17:4771 , piece_id=EKO76OTJ3IOQCDF24AQ4RYBGGHY4SCREYKJKGDFRIOQBZVHD3ISA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:49420->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761860874 (187h18m32.442056433s) [noise=true , address=45.91.116.11:4119 , piece_id=O2ZRRTZA3PE2JE5CZIURS6YB2UE6BRLZYSHHWI25GQUXQ6M42HWA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:49784->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761926208 (187h18m32.442121778s) [noise=true , address=77.90.31.14:4661 , piece_id=H33CIU52RM7ULRVA6TPCGUBNZO7WRZU3X2FBLG5U42MXDKKVEBLA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: read tcp 188.166.225.170:54942->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.761942390 (187h18m32.442137949s) [upload=164850]: Next segment err: context deadline exceeded
Sep 6 09:26:35.762108017 (187h18m32.442303586s) [noise=true , address=45.91.116.12:4227 , piece_id=JERDILAP2VBG7BBKCZFVBPA4FAAITBMRVPN3KQ27RG245ZL54C6A, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:40800->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762150447 (187h18m32.442346016s) [noise=true , address=45.91.116.14:4191 , piece_id=NC7UWT6MQJX5FEVZWZ7TUH3VET3HRD3W3GKXLPAFF2YLPKD4E2FA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:47720->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762184113 (187h18m32.442379682s) [noise=true , address=45.91.116.6:4975 , piece_id=YDMLTKJNXX6GGLVJJYGRNSKDVU7KCOACKHIVZQSOI47VROI4RATA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:59948->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762191737 (187h18m32.442387306s) [noise=true , address=77.90.31.12:4590 , piece_id=7OVJDO6JKOOPYPIZJZGAD4XFWTRWS6IPQ66UKQYEUS5DWGCMHXZQ, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 188.166.225.170:58734->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762199131 (187h18m32.4423947s) [noise=true , address=77.90.31.14:4662 , piece_id=4QDVIDURB7AOZLX6F6J7YHXTSOIPYUYKRTUSV62XCSXGNWC5YBNA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 188.166.225.170:48358->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762213709 (187h18m32.442409268s) [noise=true , address=45.91.116.6:4974 , piece_id=5K2SEZ4B7HOUXLQJ4DODNPPCXMKUPJIGNS3F2TOY56MYPSOWYZAA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:40776->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762289794 (187h18m32.442485353s) [noise=true , address=77.90.31.16:4735 , piece_id=RRXXNSLZ5VJUSGS6LET2UZ2AKUBU77DYA6DBVMKGWL75W2SN5ZGA, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 188.166.225.170:43344->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762320273 (187h18m32.442515832s) [noise=true , address=45.91.116.8:4047 , piece_id=SV2WQI7MVKWBPHQ3QME6HL3JGRIEIKDEBCOJN7BBC7VOUF2NIH4Q, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:44822->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762326685 (187h18m32.442522254s) [noise=true , address=45.91.116.7:4011 , piece_id=HZFXXZ2XEKL7CJKA6RRJGPHC2WVI2QN5IOZJPLHGJXNMPL56JENQ, seg_pos={0 0}, upload=164850]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: read tcp 188.166.225.170:50970->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:26:35.762345020 (187h18m32.442540589s) [seg_pos={0 0}, upload=164850]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil>
Duration: 31.390139318s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.000382014s
FAIL (2025-09-06 09:26:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000413805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.001041732s
FAIL (2025-09-06 09:26:27 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 30.001070497s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
3.602259949s
FAIL (2025-09-06 09:26:24 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 3.602278594s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.027684045s
FAIL (2025-09-06 09:26:21 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 30.027709874s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.000574128s
FAIL (2025-09-06 09:26:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000590258s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.000695545s
FAIL (2025-09-06 09:26:05 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 30.000731344s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.697872344s
FAIL (2025-09-06 09:25:54 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 5.697909144s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.17282204s
FAIL (2025-09-06 09:25:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 11)
======LOGS======
1.147:4016 , piece_id=A7G7RR75ILRYXWTRBJ46XDW4XZCQE4MOI7L74FL25KIQBDCQMEXQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: <nil>
Sep 6 09:24:46.612329728 (187h16m43.292525297s) [noise=true , address=89.149.201.147:4007 , piece_id=R6KIOHCLUQFEED6BSUZIJH3UARURRBMSEYOMTUO5PUPLX757IAFA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: <nil>
Sep 6 09:25:43.968041577 (187h17m40.648237146s) [noise=true , address=81.171.14.70:4025 , piece_id=I6R4LEFMLR5SB2RLVX5X4L23HYZHOIDMZKC3WMNKE2QTJWMFB42A, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, address:81.171.14.70:4025): protocol: write tcp 188.166.225.170:38182->81.171.14.70:4025: use of closed network connection; write tcp 188.166.225.170:38182->81.171.14.70:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38182->81.171.14.70:4025: use of closed network connection
Sep 6 09:25:43.968108405 (187h17m40.648303974s) [noise=true , address=81.171.14.70:4045 , piece_id=WZLQMZIXH6SXXELF2Q4UNB52GGG6BDQHPIQVVZIJJLFL62B7DAYA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12WCMeEEkwoqbPeyENfJt6nahQz2h3yW7st1TtvoY6KDKZqUGmS, address:81.171.14.70:4045): protocol: read tcp 188.166.225.170:59954->81.171.14.70:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968225068 (187h17m40.648420627s) [noise=true , address=185.167.204.205:4022 , piece_id=BJYK3X5WSKLHJFXB4AKDKU2BKMXTW2IW65NRIWJ574NP4SNFWZSA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:35842->185.167.204.205:4022: use of closed network connection; write tcp 188.166.225.170:35842->185.167.204.205:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35842->185.167.204.205:4022: use of closed network connection
Sep 6 09:25:43.968289041 (187h17m40.64848462s) [noise=true , address=81.171.14.70:4040 , piece_id=LE4JTSOFR7IKMNARFM3T6GNKKN2G4XMV5FUTESFJVKIZCFAUBHEQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:55652->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:55652->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55652->81.171.14.70:4040: use of closed network connection
Sep 6 09:25:43.968521134 (187h17m40.648716704s) [noise=true , address=185.167.204.206:4054 , piece_id=4RX3QLN7IMQK7WZ4QVINOMPWDIRBQIFMEXBPDIHSYD7B55ZIJUXQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: write tcp 188.166.225.170:45064->185.167.204.206:4054: use of closed network connection; write tcp 188.166.225.170:45064->185.167.204.206:4054: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45064->185.167.204.206:4054: use of closed network connection
Sep 6 09:25:43.968587411 (187h17m40.64878298s) [noise=true , address=81.171.14.70:4043 , piece_id=64NB7K2A7F2JO6NISJV6AY5PSHTEEABMDD3YMUXJZINAPPNSBCSA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: write tcp 188.166.225.170:33602->81.171.14.70:4043: use of closed network connection; write tcp 188.166.225.170:33602->81.171.14.70:4043: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33602->81.171.14.70:4043: use of closed network connection
Sep 6 09:25:43.968611697 (187h17m40.648807257s) [noise=true , address=185.167.204.206:4061 , piece_id=L7RQ4A276CH3BEL27LSJMQGEZUZK5RHQUH6BAPWNASJWDYKCF3DA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1G18tJ6BsYF3A1sLpRTTGzomqRWsRsj8tbHMJQeR1tZunoTDia, address:185.167.204.206:4061): protocol: read tcp 188.166.225.170:38556->185.167.204.206:4061: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968628811 (187h17m40.64882438s) [noise=true , address=81.171.14.70:4026 , piece_id=T3D3QSRRP7BD7SQRM67BFNSCA4KKCD4OD6HVTJVOYCI3Q4YNMOYA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:32830->81.171.14.70:4026: use of closed network connection; write tcp 188.166.225.170:32830->81.171.14.70:4026: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32830->81.171.14.70:4026: use of closed network connection
Sep 6 09:25:43.968650883 (187h17m40.648846452s) [noise=true , address=81.171.14.70:4035 , piece_id=TKFNGBP4JI5FT6BX5RHQKBVOZ4ICG7WLUHCTIKDUTQD76BJRR74A, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:41598->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968671893 (187h17m40.648867463s) [noise=true , address=81.171.14.70:4032 , piece_id=BEKM43VWTZAGYUUTNZSIJBJTBOLQ2NHBCPWZABFBET4GG3ZQYSFA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf, address:81.171.14.70:4032): protocol: write tcp 188.166.225.170:58030->81.171.14.70:4032: use of closed network connection; write tcp 188.166.225.170:58030->81.171.14.70:4032: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58030->81.171.14.70:4032: use of closed network connection
Sep 6 09:25:43.968688745 (187h17m40.648884304s) [noise=true , address=185.167.204.206:4050 , piece_id=QHYPN477ECDGMVA6EPNEJ4TKMVDJJHAL4LVELPEIYRJ7TD6GHG5Q, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: read tcp 188.166.225.170:45236->185.167.204.206:4050: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968742628 (187h17m40.648938217s) [noise=true , address=81.171.14.70:4039 , piece_id=LDI4R5SNFCDELQF4E6PMII7SIG6S24UCSOZXX5COVI3EN6WZPJXQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1qsAHeMX7hadREKwHeJGrLGBCVjc223xFbZ53943A2RvJnSVuC, address:81.171.14.70:4039): protocol: read tcp 188.166.225.170:35536->81.171.14.70:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968832330 (187h17m40.649027899s) [noise=true , address=185.167.204.206:4055 , piece_id=NGNCMHHTWDQ7K37OY6FYWECJ4PLIHYDAHZSBVJ554WS5GFO3VRZQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: read tcp 188.166.225.170:57968->185.167.204.206:4055: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968877426 (187h17m40.649072985s) [noise=true , address=185.167.204.206:4052 , piece_id=S3OTRWNBGBRZJWDLY4LQH5OP2VCT5G52ODPOWNSATOM2YQISHKOQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: read tcp 188.166.225.170:53744->185.167.204.206:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968910670 (187h17m40.649106249s) [noise=true , address=185.167.204.205:4013 , piece_id=FFZYATYS2YTKN2ML2QLBRG4ACWBCM2BYXHKQ4MS2H2GE6L5OZNPQ, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: read tcp 188.166.225.170:46256->185.167.204.205:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.968972929 (187h17m40.649168499s) [noise=true , address=81.171.14.70:4038 , piece_id=UEA56AOXB32ZIGFQ6NCKI7QTJA2E2OWL47NVU2AE2PHENNRT56QA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: read tcp 188.166.225.170:60462->81.171.14.70:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.969028716 (187h17m40.649224335s) [noise=true , address=185.167.204.205:4035 , piece_id=G746BEFF5DWNISVZE3NEH332NCEN4LE22EFZ44PIXT2NA42AOWOA, seg_pos={0 0}, upload=164834]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: read tcp 188.166.225.170:43700->185.167.204.205:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:43.969076417 (187h17m40.649272016s) [seg_pos={0 0}, upload=164834]: Done waiting for segment. successful: 11 optimal: 26 errs: <nil>
Sep 6 09:25:44.139379220 (187h17m40.819574789s) [upload=164834]: Done uploading segments...
Sep 6 09:25:44.139405320 (187h17m40.819600889s) [upload=164834]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 11)
Duration: 1m0.1728532s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.633439021s
FAIL (2025-09-06 09:25:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0)
======LOGS======
3K7LEU4FD73DSUKYUU5ZPHOIB7JBPUTAXB27S4USQA, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.602221930 (187h17m19.282417489s) [noise=true , address=45.91.116.6:4974 , piece_id=MNM7YSG637XGFY42LJRC5N34NLXGOZLHUUUSR2PVT7YJGU2QP3YQ, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.602480135 (187h17m19.282675704s) [noise=true , address=45.91.116.11:4119 , piece_id=3DXUAPWJ3PQLBXPT5WY7Y2INXMVHYDXDG2OJYLHDG5XERBQEW7XA, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.602509291 (187h17m19.28270486s) [noise=true , address=45.91.116.14:4189 , piece_id=DLYMGZPF3K3NNOZGSBMC34BU5S2XP7HTRJBKGWABUU436K2MUUVQ, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.602700777 (187h17m19.282896346s) [noise=true , address=45.91.116.11:4117 , piece_id=FFE7YRRRKD3CTQB7KXZWVBX6T4D5PDRLHHOWXARQ5HGQ6H25KWRQ, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.602881322 (187h17m19.283076891s) [noise=true , address=45.91.116.9:4261 , piece_id=LLSOJIPUADZ6VO4EUJVL6ZW4VIEIMDAPHMPO5RVECQCLULEFB7AQ, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.602924084 (187h17m19.283119653s) [noise=true , address=77.90.31.12:4590 , piece_id=YOXOMIQRYPA3UEVWEFT5H4OCC4NBEQPAEGWLN2GKSTTPUQJ6AMGA, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.603139256 (187h17m19.283334825s) [noise=true , address=45.91.116.12:4227 , piece_id=RHQJ74WAQBYOYRHN3DIANQ34TEAJDX2MDJOXBXV5S6B3ISYSMUGQ, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:22.603183270 (187h17m19.283378839s) [noise=true , address=77.90.31.14:4661 , piece_id=AVX4KTLVHJVWFFORYEKITRNZZBWCLSIXKH6IIIFYLWN5L5MD743Q, seg_pos={0 0}, upload=164840]: Uploading piece...
Sep 6 09:25:41.130910959 (187h17m37.811106528s) [upload=164840]: Next segment err: context deadline exceeded
Sep 6 09:25:41.131403883 (187h17m37.811599452s) [noise=true , address=45.91.116.13:4155 , piece_id=XSIOBC2AMR5T3FCT3JIWJLRR4EYWBYHHWP3EW53A4XRNFKJCVQTA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 188.166.225.170:57622->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:57622->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57622->45.91.116.13:4155: use of closed network connection
Sep 6 09:25:41.131554881 (187h17m37.81175045s) [noise=true , address=45.91.116.12:4225 , piece_id=EGYATODP2R2ZLU56GUEHV6HMNSFJSGCGTHJOZYIHIL62Y4DPDJMA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:34012->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.131761096 (187h17m37.811956665s) [noise=true , address=77.90.31.14:4661 , piece_id=AVX4KTLVHJVWFFORYEKITRNZZBWCLSIXKH6IIIFYLWN5L5MD743Q, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:44310->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:44310->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44310->77.90.31.14:4661: use of closed network connection
Sep 6 09:25:41.131830799 (187h17m37.812026368s) [noise=true , address=77.90.31.21:4878 , piece_id=CDBER5YVB63BRCIC662LPKJBYX4MDX2YIYUKJJG5YMRCZ73Y3C4Q, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 188.166.225.170:34402->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132012878 (187h17m37.812208437s) [noise=true , address=45.91.116.8:4047 , piece_id=MCND7Z4VNS3K7LEU4FD73DSUKYUU5ZPHOIB7JBPUTAXB27S4USQA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:37860->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132073243 (187h17m37.812268822s) [noise=true , address=45.91.116.11:4119 , piece_id=3DXUAPWJ3PQLBXPT5WY7Y2INXMVHYDXDG2OJYLHDG5XERBQEW7XA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40132->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132130713 (187h17m37.812326282s) [noise=true , address=45.91.116.9:4261 , piece_id=LLSOJIPUADZ6VO4EUJVL6ZW4VIEIMDAPHMPO5RVECQCLULEFB7AQ, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: read tcp 188.166.225.170:58420->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132154929 (187h17m37.812350488s) [noise=true , address=45.91.116.14:4191 , piece_id=HSBB2N6GVZACOGKKRL6GWS2ZUS4URDWXSAL2T6RXGFFYVRJUTH6A, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:41614->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132178044 (187h17m37.812373613s) [noise=true , address=45.91.116.12:4227 , piece_id=RHQJ74WAQBYOYRHN3DIANQ34TEAJDX2MDJOXBXV5S6B3ISYSMUGQ, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:52318->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132227859 (187h17m37.812423428s) [noise=true , address=45.91.116.14:4189 , piece_id=DLYMGZPF3K3NNOZGSBMC34BU5S2XP7HTRJBKGWABUU436K2MUUVQ, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: read tcp 188.166.225.170:52296->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132276563 (187h17m37.812472142s) [noise=true , address=77.90.31.18:4807 , piece_id=UKSZKCSBCU4Y7AMLFBA6JZ4K42N7DZ5RG3GZKRUKZLGOMY7FYVEA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:42158->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132323532 (187h17m37.812519101s) [noise=true , address=77.90.31.12:4590 , piece_id=YOXOMIQRYPA3UEVWEFT5H4OCC4NBEQPAEGWLN2GKSTTPUQJ6AMGA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 188.166.225.170:57534->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132333662 (187h17m37.812529231s) [noise=true , address=45.91.116.4:4939 , piece_id=XNALPWQ5AQWLWL5UEIILLTIUSH2INKNABU4OATIV42UI642HRCIA, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:60198->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132364370 (187h17m37.812559929s) [noise=true , address=45.91.116.11:4117 , piece_id=FFE7YRRRKD3CTQB7KXZWVBX6T4D5PDRLHHOWXARQ5HGQ6H25KWRQ, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 188.166.225.170:39012->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132394378 (187h17m37.812589947s) [noise=true , address=45.91.116.6:4974 , piece_id=MNM7YSG637XGFY42LJRC5N34NLXGOZLHUUUSR2PVT7YJGU2QP3YQ, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:50232->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132417773 (187h17m37.812613342s) [noise=true , address=45.91.116.14:4190 , piece_id=JO3DBO55OPJH4RJDZATLDHDP6TSBKPBOZ4G7G5L4RPJQZLGGCT5A, seg_pos={0 0}, upload=164840]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:54788->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 6 09:25:41.132445756 (187h17m37.812641325s) [seg_pos={0 0}, upload=164840]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil>
Duration: 30.633451985s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.870503065s
FAIL (2025-09-06 09:25:38 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 5.870530938s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.000396434s
FAIL (2025-09-06 09:25:35 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 30.000421762s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.00098659s
FAIL (2025-09-06 09:25:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00100729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001064092s
FAIL (2025-09-06 09:25:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001087998s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
19.471711072s
FAIL (2025-09-06 09:25:25 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 19.471742823s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001351021s
FAIL (2025-09-06 09:25:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001370728s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.178607752s
FAIL (2025-09-06 09:25:16 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 49 but got 46)
Duration: 30.178630105s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
54.746650184s
FAIL (2025-09-06 09:25:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Sep 6 09:24:16.846657430 (187h16m13.526853039s) [upload=164828]: Starting upload
Sep 6 09:24:16.846778631 (187h16m13.52697421s) [upload=164828]: Done starting upload
Sep 6 09:24:16.846876038 (187h16m13.527071607s) [upload=164828]: Uploading segments...
Sep 6 09:24:16.847102521 (187h16m13.52729809s) [upload=164828]: Got next segment. Inline: false
Sep 6 09:24:16.847153609 (187h16m13.527349168s) [upload=164828]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 09:24:16.847158668 (187h16m13.527354237s) [upload=164828]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 09:24:17.110825788 (187h16m13.791021357s) [seg_pos={0 0}, upload=164828]: Begin upload segment...
Sep 6 09:24:17.110866236 (187h16m13.791061805s) [seg_pos={0 0}, upload=164828]: Done begin upload segment.
Sep 6 09:24:17.110892877 (187h16m13.791088446s) [noise=true , address=45.91.116.6:4975 , piece_id=3HN5CO2QNFURXPN7U3XZCWXLKMX4TL3KUJJDNAIUMDRMBX4DWCLQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.110910450 (187h16m13.791106019s) [noise=true , address=45.91.116.6:4973 , piece_id=4AGRYGGPVV3SVP3ME4AVVSDRAEEXA72T3VEEUW6WE3ZPIQKQBDGA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.111298112 (187h16m13.791493681s) [noise=true , address=77.90.31.10:4520 , piece_id=SJJ7U6UQCP3FSHN5FH2DEZQ5GF23XE4AVROB62B5VOKB7ZBCCRNA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.111390940 (187h16m13.791586509s) [noise=true , address=77.90.31.21:4878 , piece_id=3JU2FZDLULOI3QU6YEWATQWFYFE5T3ERA3MESDZOJNAQ7NERNKUA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.111592936 (187h16m13.791788505s) [noise=true , address=77.90.31.19:4843 , piece_id=CNHPEQYK7YBZPMJOQUTARQELWNCWWDFX7HZJ6TPJMPGQXG73PWTA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.111787388 (187h16m13.791982957s) [noise=true , address=45.91.116.5:4298 , piece_id=5E7UQPYOP3SRCZOAEYIWCNLKTN4UHA35XXKTISU6AWVXRFAOU5VA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.111868363 (187h16m13.792063922s) [noise=true , address=45.91.116.11:4118 , piece_id=WUYVFJUYIE3OQDQ7MENYRE2T3GC3TLEERVV67WG5742PQFSXZIXA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.112134332 (187h16m13.792329891s) [noise=true , address=45.91.116.10:4083 , piece_id=VWBXZYGPVZQ6JDKP3TFAJ3NUBIJIJK3T3QNEPUFPDC2GLFZ23BWA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.112396855 (187h16m13.792592414s) [noise=true , address=45.91.116.14:4190 , piece_id=V3WSXPSZBVF4LP5KPVBAISTHIJEB34JU7ETS4SDC6JSCORP65RLA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.113241341 (187h16m13.79343693s) [noise=true , address=45.91.116.4:4938 , piece_id=N4CWKFUDXDVE47RD6FY2LL73C6T3QCDJRQPEADBCFN33T2EGW6HQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.113691873 (187h16m13.793887442s) [noise=true , address=45.91.116.8:4048 , piece_id=TFQ7ONXCFYMIAWFX2FGRSAMTUC5NVLPSAUFDVOK2CDU2KADGJOYA, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.118162267 (187h16m13.798357836s) [noise=true , address=45.91.116.11:4119 , piece_id=AQY3S6J5S3ZAWBTGUKY5PWBYBYEM46W3L65CZYV7ALFGQ5DJAILQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.118523719 (187h16m13.798719288s) [noise=true , address=45.91.116.8:4047 , piece_id=PWLCVSLDB3JOR4IAC2JKMYOLXGJWXGA7VLK7QY4CF4KNFVBN2BYQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.118946788 (187h16m13.799142357s) [noise=true , address=45.91.116.5:4297 , piece_id=KCU74HLSZBBQINE5Q3PKICJKZ4ERXCKAPD4L65IYWGTXL5RJSSUQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.119380849 (187h16m13.799576428s) [noise=true , address=77.90.31.14:4663 , piece_id=KTF5VG3AYYJFLR2Q33Q5YAZYXXYZCCDFT7TWDPJVE6ECEHB5WPIQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:17.119853333 (187h16m13.800048983s) [noise=true , address=45.91.116.9:4262 , piece_id=IA4C2Q4AG5WOAX3FDOG7IDENH5Z5NUGE7OCU67OBAN6I7N5QQ5SQ, seg_pos={0 0}, upload=164828]: Uploading piece...
Sep 6 09:24:18.923908108 (187h16m15.604103668s) [upload=164828]: Next returned nil segment
Sep 6 09:24:18.923945279 (187h16m15.604140838s) [upload=164828]: Waiting for error group managing segments...
Sep 6 09:24:19.211012246 (187h16m15.891207815s) [noise=true , address=45.91.116.14:4190 , piece_id=V3WSXPSZBVF4LP5KPVBAISTHIJEB34JU7ETS4SDC6JSCORP65RLA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.263968931 (187h16m15.9441645s) [noise=true , address=77.90.31.19:4843 , piece_id=CNHPEQYK7YBZPMJOQUTARQELWNCWWDFX7HZJ6TPJMPGQXG73PWTA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.267087980 (187h16m15.947283549s) [noise=true , address=45.91.116.6:4973 , piece_id=4AGRYGGPVV3SVP3ME4AVVSDRAEEXA72T3VEEUW6WE3ZPIQKQBDGA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.267200065 (187h16m15.947395624s) [noise=true , address=45.91.116.8:4047 , piece_id=PWLCVSLDB3JOR4IAC2JKMYOLXGJWXGA7VLK7QY4CF4KNFVBN2BYQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.270320666 (187h16m15.950516225s) [noise=true , address=45.91.116.11:4119 , piece_id=AQY3S6J5S3ZAWBTGUKY5PWBYBYEM46W3L65CZYV7ALFGQ5DJAILQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.284155861 (187h16m15.96435143s) [noise=true , address=45.91.116.10:4083 , piece_id=VWBXZYGPVZQ6JDKP3TFAJ3NUBIJIJK3T3QNEPUFPDC2GLFZ23BWA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.286768159 (187h16m15.966963728s) [noise=true , address=45.91.116.9:4262 , piece_id=IA4C2Q4AG5WOAX3FDOG7IDENH5Z5NUGE7OCU67OBAN6I7N5QQ5SQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.292397601 (187h16m15.97259317s) [noise=true , address=45.91.116.4:4938 , piece_id=N4CWKFUDXDVE47RD6FY2LL73C6T3QCDJRQPEADBCFN33T2EGW6HQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.310423705 (187h16m15.990619274s) [noise=true , address=45.91.116.6:4975 , piece_id=3HN5CO2QNFURXPN7U3XZCWXLKMX4TL3KUJJDNAIUMDRMBX4DWCLQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.311337664 (187h16m15.991533233s) [noise=true , address=45.91.116.5:4298 , piece_id=5E7UQPYOP3SRCZOAEYIWCNLKTN4UHA35XXKTISU6AWVXRFAOU5VA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.314550551 (187h16m15.99474612s) [noise=true , address=77.90.31.14:4663 , piece_id=KTF5VG3AYYJFLR2Q33Q5YAZYXXYZCCDFT7TWDPJVE6ECEHB5WPIQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.319376066 (187h16m15.999571635s) [noise=true , address=77.90.31.10:4520 , piece_id=SJJ7U6UQCP3FSHN5FH2DEZQ5GF23XE4AVROB62B5VOKB7ZBCCRNA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.382126311 (187h16m16.06232188s) [noise=true , address=77.90.31.21:4878 , piece_id=3JU2FZDLULOI3QU6YEWATQWFYFE5T3ERA3MESDZOJNAQ7NERNKUA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: <nil>
Sep 6 09:24:19.382154716 (187h16m16.062350285s) [seg_pos={0 0}, upload=164828]: Segment reached optimal threshold of 13 pieces.
Sep 6 09:24:19.382284003 (187h16m16.062479572s) [noise=true , address=45.91.116.5:4297 , piece_id=KCU74HLSZBBQINE5Q3PKICJKZ4ERXCKAPD4L65IYWGTXL5RJSSUQ, seg_pos={0 0}, upload=164828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Sep 6 09:24:19.382369908 (187h16m16.062565477s) [noise=true , address=45.91.116.11:4118 , piece_id=WUYVFJUYIE3OQDQ7MENYRE2T3GC3TLEERVV67WG5742PQFSXZIXA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Sep 6 09:24:19.382439019 (187h16m16.062634588s) [noise=true , address=45.91.116.8:4048 , piece_id=TFQ7ONXCFYMIAWFX2FGRSAMTUC5NVLPSAUFDVOK2CDU2KADGJOYA, seg_pos={0 0}, upload=164828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:41600->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:41600->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41600->45.91.116.8:4048: use of closed network connection
Sep 6 09:24:19.382465520 (187h16m16.062661089s) [seg_pos={0 0}, upload=164828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 09:24:19.382530084 (187h16m16.062725653s) [upload=164828]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 09:24:19.382533761 (187h16m16.06272933s) [upload=164828]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 09:25:10.874453050 (187h17m7.554648619s) [upload=164828]: Done uploading segments...
Sep 6 09:25:10.874479521 (187h17m7.55467509s) [upload=164828]: Upload finished. err: metaclient: context deadline exceeded
Duration: 54.746670785s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001176668s
FAIL (2025-09-06 09:25:01 UTC)
orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 6 09:24:38.250305387 (187h16m34.930500946s) [upload=164833]: Starting upload Sep 6 09:24:38.250368148 (187h16m34.930563717s) [upload=164833]: Done starting upload Sep 6 09:24:38.250417582 (187h16m34.930613151s) [upload=164833]: Uploading segments... Sep 6 09:24:38.250453882 (187h16m34.930649451s) [upload=164833]: Got next segment. Inline: true Sep 6 09:24:38.250457689 (187h16m34.930653258s) [upload=164833]: Waiting for error group managing segments... Sep 6 09:24:38.250490481 (187h16m34.93068605s) [upload=164833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 6 09:24:38.250496332 (187h16m34.930691901s) [upload=164833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Sep 6 09:24:38.250499048 (187h16m34.930694617s) [upload=164833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Sep 6 09:24:54.678052780 (187h16m51.358248349s) [upload=164833]: Done uploading segments... Sep 6 09:24:54.678076235 (187h16m51.358271874s) [upload=164833]: Upload finished. err: <nil>
Duration: 30.001200624s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000838308s
FAIL (2025-09-06 09:24:56 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.000866001s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000661042s
FAIL (2025-09-06 09:24:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000680728s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
13.572708637s
FAIL (2025-09-06 09:24:36 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 13.5727729s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.291390122s
FAIL (2025-09-06 09:09:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.291407595s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
12.007317496s
FAIL (2025-09-06 09:08:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 12.007346582s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
8.254181571s
FAIL (2025-09-06 09:07:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 8.25420702s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
12.189062458s
FAIL (2025-09-06 09:07:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 12.18916798s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.714482378s
FAIL (2025-09-06 09:06:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.714529678s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.677527325s
FAIL (2025-09-06 09:05:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.677555579s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.04422085s
FAIL (2025-09-06 09:05:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.044251949s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
11.425646602s
FAIL (2025-09-06 09:04:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 11.425679345s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.614898458s
FAIL (2025-09-06 09:03:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.614928546s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.410229493s
FAIL (2025-09-06 09:02:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.410254951s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
6.380490715s
FAIL (2025-09-06 09:01:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.38051466s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
8.232761562s
FAIL (2025-09-06 09:01:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 8.232793714s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
8.091292827s
FAIL (2025-09-06 09:00:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 8.091344685s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
7.432777217s
FAIL (2025-09-06 08:59:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 7.432802355s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.519179661s
FAIL (2025-09-06 08:59:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.519205889s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.109675623s
FAIL (2025-09-06 08:58:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.109713075s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
10.37802073s
FAIL (2025-09-06 08:56:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 10.378069024s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
9.945475943s
FAIL (2025-09-06 08:55:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 9.945503537s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.237856603s
FAIL (2025-09-06 08:55:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.23788693s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
6.437711641s
FAIL (2025-09-06 08:55:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.437739414s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.253795544s
FAIL (2025-09-06 08:54:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 6.253828559s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
10.993910501s
FAIL (2025-09-06 08:53:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 6 08:24:39.098120559 (186h16m35.778316128s) [upload=163941]: Starting upload
Sep 6 08:24:39.098254866 (186h16m35.778450445s) [upload=163941]: Done starting upload
Sep 6 08:24:39.098377831 (186h16m35.77857341s) [upload=163941]: Uploading segments...
Sep 6 08:24:39.098725336 (186h16m35.778920906s) [upload=163941]: Got next segment. Inline: false
Sep 6 08:24:39.098762338 (186h16m35.778957897s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 6 08:24:39.098767808 (186h16m35.778963437s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 6 08:24:39.399907795 (186h16m36.080103354s) [seg_pos={0 0}, upload=163941]: Begin upload segment...
Sep 6 08:24:39.399949796 (186h16m36.080145365s) [seg_pos={0 0}, upload=163941]: Done begin upload segment.
Sep 6 08:24:39.399972369 (186h16m36.080167938s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400054356 (186h16m36.080249935s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400383346 (186h16m36.080578915s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400826173 (186h16m36.081021752s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.400897349 (186h16m36.081092918s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401305220 (186h16m36.081500789s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401335588 (186h16m36.081531157s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401705696 (186h16m36.081901265s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.401839642 (186h16m36.082035211s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402029265 (186h16m36.082224834s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402273552 (186h16m36.082469122s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402334850 (186h16m36.082530419s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402658108 (186h16m36.082853677s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402667447 (186h16m36.082863016s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.402924999 (186h16m36.083120568s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:39.403024360 (186h16m36.083219929s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Uploading piece...
Sep 6 08:24:41.130554591 (186h16m37.810750161s) [upload=163941]: Next returned nil segment
Sep 6 08:24:41.130573418 (186h16m37.810768987s) [upload=163941]: Waiting for error group managing segments...
Sep 6 08:24:41.436098347 (186h16m38.116293916s) [noise=true , address=45.91.116.4:4939 , piece_id=T3Q7VQQOET5WJPNR5WT2RDB3NC2XEOYVMEROONIKEPCPUOZ3VIXQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.489189219 (186h16m38.169384778s) [noise=true , address=45.91.116.8:4048 , piece_id=4RX3OBUYFG4NNUPCPG7WZNSGVM4WXE76OQBTZCVLC7VLR4KVDYWQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.490290868 (186h16m38.170486437s) [noise=true , address=45.91.116.8:4047 , piece_id=TOSASWI7WZFT3KBUFBEOVXUVCUAKVQJMMDZZDYPCKRISRDO3ESTA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.497294599 (186h16m38.177490168s) [noise=true , address=77.90.31.10:4518 , piece_id=SXDRQV7H3YFUQNDYMHI5TVQ3HLJ3EST545NMR4WWL4PDB6KPTEZA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.502719791 (186h16m38.18291536s) [noise=true , address=77.90.31.10:4517 , piece_id=XJYMGQYHQEWRK75DUBFLWF2UZBAMCOU4QLFGC3DLYUGRADETWU5Q, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.531737216 (186h16m38.211932796s) [noise=true , address=45.91.116.8:4045 , piece_id=OIMF42HTOFHJS27PX5DONUUKSQJKSWQZPVXLNCGSGCYEZKMDL52A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.534204297 (186h16m38.214399866s) [noise=true , address=77.90.31.12:4592 , piece_id=R3S7UUBYFBAM5XOCVFFGNI55F3TOIS7KOR3JAUNA5CMC4R7WJDCA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.538312950 (186h16m38.218508519s) [noise=true , address=45.91.116.4:4938 , piece_id=NJQFCGIVVOGKHNGKYLFWGT4Z2J5KB7QQ3B3TEYOOFXW7NC2QWTAQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.539907000 (186h16m38.220102569s) [noise=true , address=45.91.116.13:4154 , piece_id=MEZMRN4BNWSDVY5LMTEWZATIH5CX3743A3R5IPW6M3UCYZYF43TA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.551283249 (186h16m38.231478808s) [noise=true , address=45.91.116.14:4190 , piece_id=FMIHC4ZPE25O4K6FR5EFD5S5TFZVRPPLKYSQVBHXBCBQDWRZVONA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.576854151 (186h16m38.25704971s) [noise=true , address=77.90.31.14:4662 , piece_id=ARNSWUJMPYY3JBMPI3IHBP3T3TA4J5RIGOJ4LQS6ZF7DBEDU5LMQ, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.580286720 (186h16m38.260482289s) [noise=true , address=77.90.31.15:4697 , piece_id=DCDCWOHUMNETJ5CYFWF3WKNI4PEXWGBOFG6WYT7OEWJ7DQZOZS3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599952070 (186h16m38.280147639s) [noise=true , address=45.91.116.6:4974 , piece_id=3JCSNNQSFGJ5I2DKYULMENXGC3E7KE6IIGPOUETQ7N5L4NQ3V3LA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: <nil>
Sep 6 08:24:41.599973702 (186h16m38.280169271s) [seg_pos={0 0}, upload=163941]: Segment reached optimal threshold of 13 pieces.
Sep 6 08:24:41.600109011 (186h16m38.28030457s) [noise=true , address=77.90.31.19:4842 , piece_id=ZBKYLVES2G2TB6TM2RKS7Q2HJKPZ6DBWPPLEAJPWEC3BEHOUBS2A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600213861 (186h16m38.28040942s) [noise=true , address=77.90.31.17:4771 , piece_id=6MDITR4RJCL53H3DREE2G5L2OBZBWJ3EW2P4FHW3JVK4E6GGC6VA, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600301389 (186h16m38.280496958s) [noise=true , address=45.91.116.10:4081 , piece_id=YPDFX2TCCDPBRCHVFTBEZ4IR2LZY2MAWBGQTNR5B6DUX7ZLTSM3A, seg_pos={0 0}, upload=163941]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Sep 6 08:24:41.600311428 (186h16m38.280506997s) [seg_pos={0 0}, upload=163941]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 6 08:24:41.600373867 (186h16m38.280569436s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 6 08:24:41.600377655 (186h16m38.280573224s) [upload=163941]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 6 08:24:41.943078361 (186h16m38.62327393s) [upload=163941]: Done uploading segments...
Sep 6 08:24:41.943099842 (186h16m38.623295411s) [upload=163941]: Upload finished. err: <nil>
Duration: 10.993938885s
core
uplink
us-select-1
|