Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.700555213s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:10:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.700583517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.70490063s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:10:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.704928441s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.154581973s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:10:08 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.154618283s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.735882641s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:09:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.735907758s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.23631356s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:09:44 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.236341634s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.895769213s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.895796375s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.017990728s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:08:41 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.018023851s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.690259998s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:08:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.690286898s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.667806215s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:08:10 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.667852193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.785618675s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:07:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.785648172s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.828476127s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:07:32 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.828502037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................X.........................] duration: 5.840527019s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:07:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.840557607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................X........................] duration: 8.21643442s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:06:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 8.216481641s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.069067679s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:06:35 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.069090082s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.963717464s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:05:41 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.963747612s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................X.......................] duration: 5.667265342s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:05:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.667321488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................X......................] duration: 5.781833152s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:05:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.781865875s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.663164863s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:04:44 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.663195181s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................................X.....................] duration: 5.759476847s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.75950475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................................X....................] duration: 5.842196666s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.842229598s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.083142762s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:04:08 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.083174993s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.736867623s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:03:25 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.736895275s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.................................X...................] duration: 5.707392437s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:03:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.70742525s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.468613122s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:02:18 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.468766464s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................................X..................] duration: 5.762881319s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:01:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.762909863s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.070205081s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:01:27 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.070234187s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.................................X.................] duration: 5.713068179s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.713102154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................................X................] duration: 5.696397941s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 18:00:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.696424563s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.764975049s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 18:00:42 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.765003603s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................................X...............] duration: 5.659343315s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:59:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.659375137s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.021176112s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:59:31 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.021266073s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.................................X..............] duration: 5.640014581s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:58:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.640046723s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.928416949s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:58:52 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.928444651s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.................................X.............] duration: 5.709962639s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:58:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.709994039s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.672888482s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:57:49 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.672916165s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.................................X............] duration: 5.828514874s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:57:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.828575501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.................................X...........] duration: 5.710794259s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:57:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.710824587s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.037592627s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:57:05 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 6.037678982s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................................X..........] duration: 5.743317405s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:55:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.743350529s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.894886719s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:55:51 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.895410597s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.761679664s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:55:05 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.761709312s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.................................X.........] duration: 5.644073864s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:54:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.644110664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................................X........] duration: 5.660119619s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.660146911s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.799559718s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:53:52 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.799589316s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................................X.......] duration: 5.728874186s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:53:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.728903413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................................X......] duration: 5.777598329s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:53:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.777626391s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.813038485s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:52:50 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.813061879s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.703055606s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:52:22 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.703083189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................................X.....] duration: 5.807103334s trace: https://jaeger.dev.storj.io/trace/3348cbaae8620fc6 FAIL (2025-12-04 17:52:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ) context deadline exceeded ======LOGS====== upload Dec 4 17:47:54.130543913 (24h52m30.601754642s) [upload=21870]: Uploading segments... Dec 4 17:47:54.130738174 (24h52m30.601948913s) [upload=21870]: Got next segment. Inline: false Dec 4 17:47:54.130777078 (24h52m30.601987807s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:47:54.130782648 (24h52m30.601993377s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:47:54.641736615 (24h52m31.112947354s) [seg_pos={0 0}, upload=21870]: Begin upload segment... Dec 4 17:47:54.641842867 (24h52m31.113053606s) [seg_pos={0 0}, upload=21870]: Done begin upload segment. Dec 4 17:47:54.641873866 (24h52m31.113084595s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642776033 (24h52m31.113986772s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.642789559 (24h52m31.114000288s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643336110 (24h52m31.114546849s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643652503 (24h52m31.114863272s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643829500 (24h52m31.115040239s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.643960999 (24h52m31.115171728s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644116897 (24h52m31.115327626s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644294675 (24h52m31.115505414s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.644689837 (24h52m31.115900576s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645119586 (24h52m31.116330325s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645251568 (24h52m31.116462297s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.645679973 (24h52m31.116890702s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646110963 (24h52m31.117321703s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646464476 (24h52m31.117675206s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:54.646612088 (24h52m31.117822827s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Uploading piece... Dec 4 17:47:56.398938048 (24h52m32.870148787s) [upload=21870]: Next returned nil segment Dec 4 17:47:56.398950933 (24h52m32.870161672s) [upload=21870]: Waiting for error group managing segments... Dec 4 17:47:56.742706640 (24h52m33.213917379s) [noise=true , address=77.90.31.11:4555 , piece_id=S5ZG6PBWJ6UWEUWCDQ5RRPHC6NY57QH3RL6ESFTEJPT2EZSFTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.752512893 (24h52m33.223723622s) [noise=true , address=77.90.31.10:4519 , piece_id=BUCQ7YYRTDKICTMOY2K5HGPXO2YXMMMBM7EKAJGH33TRI44Z6FWA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.778408093 (24h52m33.249618832s) [noise=true , address=145.79.227.10:4402 , piece_id=GBORMXMV7SQ37MABF2HA67TQGFDLUO7XUIPL3K7VFS6PMBPANJVQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.781030807 (24h52m33.252241546s) [noise=true , address=145.79.227.32:4836 , piece_id=FJUWEM6W5BCBH7AXYA65WOZGPO3MYPIHIJ6A3T7RAFAGVQRQDA7Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.783109295 (24h52m33.254320034s) [noise=true , address=145.79.227.8:4330 , piece_id=JWNUPIR3ANAN5NMCZYNK2EU7LFB23I64VYIBELY66BTE6KVQ23QA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.788276534 (24h52m33.259487263s) [noise=true , address=145.79.227.44:4194 , piece_id=YJQEULCLWQMBBOSICHFIJGTTMG4IWYQ2DOVJW2HTV2SZTHW6RYHA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790807554 (24h52m33.262018313s) [noise=true , address=145.79.227.39:4015 , piece_id=VNKSVQXZLZO7MTL5CBKBOC2SCSV3L3VRXJJHLUDG4YULDJJPCLCQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.790966056 (24h52m33.262176795s) [noise=true , address=145.79.227.44:4195 , piece_id=QLCRG7LM3WJ2ZLY4O26TGCZMUB6UCPUQPTM2KS3CBAK3BMII6RQA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.815931645 (24h52m33.287142374s) [noise=true , address=145.79.227.34:4905 , piece_id=AAEPN2N3UFR2IKDTPBBMC7TNCAM323PW6HBCDAAQDRDWJGJSO5SQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.828440274 (24h52m33.299651003s) [noise=true , address=145.79.227.42:4122 , piece_id=2WTGIP2SZNVTENXZCG6557IPBEKJMJKG2FX22RGW22F3PX5IE5HQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.832425682 (24h52m33.303636421s) [noise=true , address=77.90.31.21:4878 , piece_id=HDETJLESJO6KNZ7MCFYIM3LMNXCHLBPQCBZLQLVINFEUPF5DTJ4Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:56.852086020 (24h52m33.323296759s) [noise=true , address=145.79.227.27:4689 , piece_id=AHMQVLMYHCCANG3FTHRPZGSFIHHF2HFITAFWPPKZE7QWCK45D74Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119031566 (24h52m33.590242305s) [noise=true , address=145.79.227.24:4583 , piece_id=JVRVJSF5ZOWB7KLLMGRVMGK25AI3SXY7DQ3HWRBYUNXE3MLPYHLA, seg_pos={0 0}, upload=21870]: Done uploading piece. err: <nil> Dec 4 17:47:57.119058968 (24h52m33.590269698s) [seg_pos={0 0}, upload=21870]: Segment reached optimal threshold of 13 pieces. Dec 4 17:47:57.119295740 (24h52m33.590506509s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36214->145.79.227.33:4869: use of closed network connection Dec 4 17:47:57.119306189 (24h52m33.590516928s) [noise=true , address=145.79.227.33:4869 , piece_id=4LXXSJIWLIOYFGEGL6YZEMCTVAT6SJFHP7OVKZYOICWR6CIQBMXQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119718755 (24h52m33.590929494s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Dec 4 17:47:57.119727862 (24h52m33.590938601s) [noise=true , address=145.79.227.9:4367 , piece_id=RY75MW2WBEADO7BODHX4BXS4G6AI4CZK3ONXP4LWLCUTITI77VQQ, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119863280 (24h52m33.591074029s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38448->145.79.227.14:4838: use of closed network connection Dec 4 17:47:57.119871936 (24h52m33.591082675s) [noise=true , address=145.79.227.14:4838 , piece_id=SV5ND43OBMZO3V3IPTOSK6MZ3OQNKCKWWFY36GYIIWXD52RWC25Q, seg_pos={0 0}, upload=21870]: Long tail cancellation due to optimal threshold Dec 4 17:47:57.119890642 (24h52m33.591101371s) [seg_pos={0 0}, upload=21870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:47:57.119979120 (24h52m33.591189849s) [upload=21870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:47:57.119987697 (24h52m33.591198436s) [upload=21870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:47:57.440867013 (24h52m33.912077742s) [upload=21870]: Done uploading segments... Dec 4 17:47:57.440884847 (24h52m33.912095577s) [upload=21870]: Upload finished. err: <nil> Duration: 5.807131197s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.857598547s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:51:19 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.857623875s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)