Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX.] duration: 6.10553823s trace: https://jaeger.dev.storj.io/trace/6281b6dc0e613506 FAIL (2025-12-26 17:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VEHQFJIQVIJ7RUPYULBCITVVZH7XY3CRQZN2DLVPOI2V2C227SJQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: E4WEQQAYTSAPNVSZFTLMUEEWHDHX2KSL4XPSETVYUP2KZ7VE7JRQ) context deadline exceeded ======LOGS====== Dec 26 17:26:34.781608262 (505h57m34.652691297s) [upload=443967]: Starting upload Dec 26 17:26:34.781699215 (505h57m34.65278226s) [upload=443967]: Done starting upload Dec 26 17:26:34.781861453 (505h57m34.652944498s) [upload=443967]: Uploading segments... Dec 26 17:26:34.782123743 (505h57m34.653206788s) [upload=443967]: Got next segment. Inline: false Dec 26 17:26:34.782195550 (505h57m34.653278605s) [upload=443967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:26:34.782203856 (505h57m34.653286901s) [upload=443967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:26:35.031421719 (505h57m34.902504764s) [seg_pos={0 0}, upload=443967]: Begin upload segment... Dec 26 17:26:35.031478227 (505h57m34.902561262s) [seg_pos={0 0}, upload=443967]: Done begin upload segment. Dec 26 17:26:35.031500048 (505h57m34.902583094s) [noise=true , address=45.91.116.7:4011 , piece_id=R7YBESBQ4ET2UCWSOPHWDJUXRWBAROX2JXBBNBO2UGCQBGJKWHOQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.031509417 (505h57m34.902592452s) [noise=true , address=145.79.227.17:4970 , piece_id=BDF3O7WR2NF2NNQN3PHXXOXWAOQAMKOBY7DDZA7IORZXOBXC2HHA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.032153102 (505h57m34.903236147s) [noise=true , address=145.79.227.51:4411 , piece_id=JOQIMFOEMOCEFR6XSYPRDLTYWJ5RZ4BHFGXEAY6435G5QUFYVE5A, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.032619160 (505h57m34.903702195s) [noise=true , address=145.79.227.47:4265 , piece_id=7GPPD6B6EHBQIWVRB33WETQZHHXHWC3QZFAYTYCLOBOASJCZFGAA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.033020243 (505h57m34.904103278s) [noise=true , address=77.90.31.10:4517 , piece_id=E4WEQQAYTSAPNVSZFTLMUEEWHDHX2KSL4XPSETVYUP2KZ7VE7JRQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.033405998 (505h57m34.904489033s) [noise=true , address=145.79.227.4:4186 , piece_id=YI4GQAF45MYMT5R7MA2HPUJB7BRBIRNSNNHUMHLDTIPQWLOL7ECA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.033790600 (505h57m34.904873645s) [noise=true , address=145.79.227.30:4761 , piece_id=QIR3IHTQEWMTQABBHAGIBEM4PJKJVCIPTXNQH3IRH2TDXZ535BRQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.034197846 (505h57m34.905280881s) [noise=true , address=45.91.116.4:4937 , piece_id=XR7CRKUKP5WUGNX277OUDKOZUBLQBGAATYSDOSIQ2BE7F45RPO2Q, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.034756289 (505h57m34.905839324s) [noise=true , address=45.91.116.6:4975 , piece_id=VEHQFJIQVIJ7RUPYULBCITVVZH7XY3CRQZN2DLVPOI2V2C227SJQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.035240962 (505h57m34.906324007s) [noise=true , address=45.91.116.9:4264 , piece_id=5CLOEFGQUXXEDLROR7HZB4BSIPXEFKBVFLQHRK2WKYMYBXCVK7QA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.036702796 (505h57m34.907785831s) [noise=true , address=145.79.227.29:4942 , piece_id=O6DPP7ICROICHIACUAIHEJNCQ4ZZV5BWO7Z2ODXRKH6JFFMLX6JA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.038353478 (505h57m34.909436523s) [noise=true , address=145.79.227.14:4839 , piece_id=ABNZJKFBCM2LP5HBHEGGPXKJYZJKIZ2HHCV5ZHBEPATNYNCKOTJA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.039854708 (505h57m34.910937753s) [noise=true , address=145.79.227.50:4373 , piece_id=OYJIK4XZLV2VP4RD5SGCMHGPZC73MLSNDKNNW5VCVKTYT73RUQXQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.041543433 (505h57m34.912626478s) [noise=true , address=145.79.227.12:4765 , piece_id=IZBZD4PRAU3LJTTZOGL3JWU24HDLOUWVK7QAEQU65SB2OTQFWLVA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.043148359 (505h57m34.914231404s) [noise=true , address=145.79.227.39:4015 , piece_id=UDGL5UCYOXAYXKGIS6YPGJT6V4UDDDVWUIVYSBTZHJOTGDGX2MNQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.044433777 (505h57m34.915516822s) [noise=true , address=145.79.227.16:4911 , piece_id=ZMFPYYA37ME3A3WC3YUGHCK6Y4MTRZO3NLQ27BTHBAPELOUKQT4Q, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:36.652140090 (505h57m36.523223125s) [upload=443967]: Next returned nil segment Dec 26 17:26:36.652153566 (505h57m36.523236601s) [upload=443967]: Waiting for error group managing segments... Dec 26 17:26:37.038184526 (505h57m36.909267571s) [noise=true , address=145.79.227.51:4411 , piece_id=JOQIMFOEMOCEFR6XSYPRDLTYWJ5RZ4BHFGXEAY6435G5QUFYVE5A, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.052148746 (505h57m36.923231781s) [noise=true , address=145.79.227.50:4373 , piece_id=OYJIK4XZLV2VP4RD5SGCMHGPZC73MLSNDKNNW5VCVKTYT73RUQXQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.052628209 (505h57m36.923711255s) [noise=true , address=145.79.227.4:4186 , piece_id=YI4GQAF45MYMT5R7MA2HPUJB7BRBIRNSNNHUMHLDTIPQWLOL7ECA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.055711881 (505h57m36.926794926s) [noise=true , address=145.79.227.29:4942 , piece_id=O6DPP7ICROICHIACUAIHEJNCQ4ZZV5BWO7Z2ODXRKH6JFFMLX6JA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.057856214 (505h57m36.928939259s) [noise=true , address=145.79.227.39:4015 , piece_id=UDGL5UCYOXAYXKGIS6YPGJT6V4UDDDVWUIVYSBTZHJOTGDGX2MNQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.059884526 (505h57m36.930967571s) [noise=true , address=145.79.227.12:4765 , piece_id=IZBZD4PRAU3LJTTZOGL3JWU24HDLOUWVK7QAEQU65SB2OTQFWLVA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.079913095 (505h57m36.95099613s) [noise=true , address=145.79.227.47:4265 , piece_id=7GPPD6B6EHBQIWVRB33WETQZHHXHWC3QZFAYTYCLOBOASJCZFGAA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.091470612 (505h57m36.962553657s) [noise=true , address=145.79.227.16:4911 , piece_id=ZMFPYYA37ME3A3WC3YUGHCK6Y4MTRZO3NLQ27BTHBAPELOUKQT4Q, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.131217967 (505h57m37.002301012s) [noise=true , address=145.79.227.17:4970 , piece_id=BDF3O7WR2NF2NNQN3PHXXOXWAOQAMKOBY7DDZA7IORZXOBXC2HHA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.298206263 (505h57m37.169289298s) [noise=true , address=45.91.116.6:4975 , piece_id=VEHQFJIQVIJ7RUPYULBCITVVZH7XY3CRQZN2DLVPOI2V2C227SJQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.351719471 (505h57m37.222802506s) [noise=true , address=77.90.31.10:4517 , piece_id=E4WEQQAYTSAPNVSZFTLMUEEWHDHX2KSL4XPSETVYUP2KZ7VE7JRQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.407047596 (505h57m37.278130631s) [noise=true , address=45.91.116.4:4937 , piece_id=XR7CRKUKP5WUGNX277OUDKOZUBLQBGAATYSDOSIQ2BE7F45RPO2Q, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.464540522 (505h57m37.335623567s) [noise=true , address=145.79.227.14:4839 , piece_id=ABNZJKFBCM2LP5HBHEGGPXKJYZJKIZ2HHCV5ZHBEPATNYNCKOTJA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.464564408 (505h57m37.335647453s) [seg_pos={0 0}, upload=443967]: Segment reached optimal threshold of 13 pieces. Dec 26 17:26:37.464946585 (505h57m37.33602962s) [noise=true , address=145.79.227.30:4761 , piece_id=QIR3IHTQEWMTQABBHAGIBEM4PJKJVCIPTXNQH3IRH2TDXZ535BRQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 17:26:37.464956223 (505h57m37.336039268s) [noise=true , address=145.79.227.30:4761 , piece_id=QIR3IHTQEWMTQABBHAGIBEM4PJKJVCIPTXNQH3IRH2TDXZ535BRQ, seg_pos={0 0}, upload=443967]: Long tail cancellation due to optimal threshold Dec 26 17:26:37.465142819 (505h57m37.336225864s) [noise=true , address=45.91.116.7:4011 , piece_id=R7YBESBQ4ET2UCWSOPHWDJUXRWBAROX2JXBBNBO2UGCQBGJKWHOQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 17:26:37.465150474 (505h57m37.336233509s) [noise=true , address=45.91.116.7:4011 , piece_id=R7YBESBQ4ET2UCWSOPHWDJUXRWBAROX2JXBBNBO2UGCQBGJKWHOQ, seg_pos={0 0}, upload=443967]: Long tail cancellation due to optimal threshold Dec 26 17:26:37.465474370 (505h57m37.336557415s) [noise=true , address=45.91.116.9:4264 , piece_id=5CLOEFGQUXXEDLROR7HZB4BSIPXEFKBVFLQHRK2WKYMYBXCVK7QA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:46928->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:26:37.465483387 (505h57m37.336566432s) [noise=true , address=45.91.116.9:4264 , piece_id=5CLOEFGQUXXEDLROR7HZB4BSIPXEFKBVFLQHRK2WKYMYBXCVK7QA, seg_pos={0 0}, upload=443967]: Long tail cancellation due to optimal threshold Dec 26 17:26:37.465537150 (505h57m37.336620185s) [seg_pos={0 0}, upload=443967]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:26:37.465624507 (505h57m37.336707542s) [upload=443967]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:26:37.465630849 (505h57m37.336713884s) [upload=443967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:26:37.770939229 (505h57m37.642022264s) [upload=443967]: Done uploading segments... Dec 26 17:26:37.770964638 (505h57m37.642047673s) [upload=443967]: Upload finished. err: <nil> Duration: 6.105555171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX] duration: 30.004251047s trace: https://jaeger.dev.storj.io/trace/6281b6dc0e613506 FAIL (2025-12-26 17:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VEHQFJIQVIJ7RUPYULBCITVVZH7XY3CRQZN2DLVPOI2V2C227SJQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: E4WEQQAYTSAPNVSZFTLMUEEWHDHX2KSL4XPSETVYUP2KZ7VE7JRQ) context deadline exceeded ======LOGS====== Dec 26 17:26:34.781608262 (505h57m34.652691297s) [upload=443967]: Starting upload Dec 26 17:26:34.781699215 (505h57m34.65278226s) [upload=443967]: Done starting upload Dec 26 17:26:34.781861453 (505h57m34.652944498s) [upload=443967]: Uploading segments... Dec 26 17:26:34.782123743 (505h57m34.653206788s) [upload=443967]: Got next segment. Inline: false Dec 26 17:26:34.782195550 (505h57m34.653278605s) [upload=443967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:26:34.782203856 (505h57m34.653286901s) [upload=443967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:26:35.031421719 (505h57m34.902504764s) [seg_pos={0 0}, upload=443967]: Begin upload segment... Dec 26 17:26:35.031478227 (505h57m34.902561262s) [seg_pos={0 0}, upload=443967]: Done begin upload segment. Dec 26 17:26:35.031500048 (505h57m34.902583094s) [noise=true , address=45.91.116.7:4011 , piece_id=R7YBESBQ4ET2UCWSOPHWDJUXRWBAROX2JXBBNBO2UGCQBGJKWHOQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.031509417 (505h57m34.902592452s) [noise=true , address=145.79.227.17:4970 , piece_id=BDF3O7WR2NF2NNQN3PHXXOXWAOQAMKOBY7DDZA7IORZXOBXC2HHA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.032153102 (505h57m34.903236147s) [noise=true , address=145.79.227.51:4411 , piece_id=JOQIMFOEMOCEFR6XSYPRDLTYWJ5RZ4BHFGXEAY6435G5QUFYVE5A, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.032619160 (505h57m34.903702195s) [noise=true , address=145.79.227.47:4265 , piece_id=7GPPD6B6EHBQIWVRB33WETQZHHXHWC3QZFAYTYCLOBOASJCZFGAA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.033020243 (505h57m34.904103278s) [noise=true , address=77.90.31.10:4517 , piece_id=E4WEQQAYTSAPNVSZFTLMUEEWHDHX2KSL4XPSETVYUP2KZ7VE7JRQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.033405998 (505h57m34.904489033s) [noise=true , address=145.79.227.4:4186 , piece_id=YI4GQAF45MYMT5R7MA2HPUJB7BRBIRNSNNHUMHLDTIPQWLOL7ECA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.033790600 (505h57m34.904873645s) [noise=true , address=145.79.227.30:4761 , piece_id=QIR3IHTQEWMTQABBHAGIBEM4PJKJVCIPTXNQH3IRH2TDXZ535BRQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.034197846 (505h57m34.905280881s) [noise=true , address=45.91.116.4:4937 , piece_id=XR7CRKUKP5WUGNX277OUDKOZUBLQBGAATYSDOSIQ2BE7F45RPO2Q, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.034756289 (505h57m34.905839324s) [noise=true , address=45.91.116.6:4975 , piece_id=VEHQFJIQVIJ7RUPYULBCITVVZH7XY3CRQZN2DLVPOI2V2C227SJQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.035240962 (505h57m34.906324007s) [noise=true , address=45.91.116.9:4264 , piece_id=5CLOEFGQUXXEDLROR7HZB4BSIPXEFKBVFLQHRK2WKYMYBXCVK7QA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.036702796 (505h57m34.907785831s) [noise=true , address=145.79.227.29:4942 , piece_id=O6DPP7ICROICHIACUAIHEJNCQ4ZZV5BWO7Z2ODXRKH6JFFMLX6JA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.038353478 (505h57m34.909436523s) [noise=true , address=145.79.227.14:4839 , piece_id=ABNZJKFBCM2LP5HBHEGGPXKJYZJKIZ2HHCV5ZHBEPATNYNCKOTJA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.039854708 (505h57m34.910937753s) [noise=true , address=145.79.227.50:4373 , piece_id=OYJIK4XZLV2VP4RD5SGCMHGPZC73MLSNDKNNW5VCVKTYT73RUQXQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.041543433 (505h57m34.912626478s) [noise=true , address=145.79.227.12:4765 , piece_id=IZBZD4PRAU3LJTTZOGL3JWU24HDLOUWVK7QAEQU65SB2OTQFWLVA, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.043148359 (505h57m34.914231404s) [noise=true , address=145.79.227.39:4015 , piece_id=UDGL5UCYOXAYXKGIS6YPGJT6V4UDDDVWUIVYSBTZHJOTGDGX2MNQ, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:35.044433777 (505h57m34.915516822s) [noise=true , address=145.79.227.16:4911 , piece_id=ZMFPYYA37ME3A3WC3YUGHCK6Y4MTRZO3NLQ27BTHBAPELOUKQT4Q, seg_pos={0 0}, upload=443967]: Uploading piece... Dec 26 17:26:36.652140090 (505h57m36.523223125s) [upload=443967]: Next returned nil segment Dec 26 17:26:36.652153566 (505h57m36.523236601s) [upload=443967]: Waiting for error group managing segments... Dec 26 17:26:37.038184526 (505h57m36.909267571s) [noise=true , address=145.79.227.51:4411 , piece_id=JOQIMFOEMOCEFR6XSYPRDLTYWJ5RZ4BHFGXEAY6435G5QUFYVE5A, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.052148746 (505h57m36.923231781s) [noise=true , address=145.79.227.50:4373 , piece_id=OYJIK4XZLV2VP4RD5SGCMHGPZC73MLSNDKNNW5VCVKTYT73RUQXQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.052628209 (505h57m36.923711255s) [noise=true , address=145.79.227.4:4186 , piece_id=YI4GQAF45MYMT5R7MA2HPUJB7BRBIRNSNNHUMHLDTIPQWLOL7ECA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.055711881 (505h57m36.926794926s) [noise=true , address=145.79.227.29:4942 , piece_id=O6DPP7ICROICHIACUAIHEJNCQ4ZZV5BWO7Z2ODXRKH6JFFMLX6JA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.057856214 (505h57m36.928939259s) [noise=true , address=145.79.227.39:4015 , piece_id=UDGL5UCYOXAYXKGIS6YPGJT6V4UDDDVWUIVYSBTZHJOTGDGX2MNQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.059884526 (505h57m36.930967571s) [noise=true , address=145.79.227.12:4765 , piece_id=IZBZD4PRAU3LJTTZOGL3JWU24HDLOUWVK7QAEQU65SB2OTQFWLVA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.079913095 (505h57m36.95099613s) [noise=true , address=145.79.227.47:4265 , piece_id=7GPPD6B6EHBQIWVRB33WETQZHHXHWC3QZFAYTYCLOBOASJCZFGAA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.091470612 (505h57m36.962553657s) [noise=true , address=145.79.227.16:4911 , piece_id=ZMFPYYA37ME3A3WC3YUGHCK6Y4MTRZO3NLQ27BTHBAPELOUKQT4Q, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.131217967 (505h57m37.002301012s) [noise=true , address=145.79.227.17:4970 , piece_id=BDF3O7WR2NF2NNQN3PHXXOXWAOQAMKOBY7DDZA7IORZXOBXC2HHA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.298206263 (505h57m37.169289298s) [noise=true , address=45.91.116.6:4975 , piece_id=VEHQFJIQVIJ7RUPYULBCITVVZH7XY3CRQZN2DLVPOI2V2C227SJQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.351719471 (505h57m37.222802506s) [noise=true , address=77.90.31.10:4517 , piece_id=E4WEQQAYTSAPNVSZFTLMUEEWHDHX2KSL4XPSETVYUP2KZ7VE7JRQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.407047596 (505h57m37.278130631s) [noise=true , address=45.91.116.4:4937 , piece_id=XR7CRKUKP5WUGNX277OUDKOZUBLQBGAATYSDOSIQ2BE7F45RPO2Q, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.464540522 (505h57m37.335623567s) [noise=true , address=145.79.227.14:4839 , piece_id=ABNZJKFBCM2LP5HBHEGGPXKJYZJKIZ2HHCV5ZHBEPATNYNCKOTJA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: <nil> Dec 26 17:26:37.464564408 (505h57m37.335647453s) [seg_pos={0 0}, upload=443967]: Segment reached optimal threshold of 13 pieces. Dec 26 17:26:37.464946585 (505h57m37.33602962s) [noise=true , address=145.79.227.30:4761 , piece_id=QIR3IHTQEWMTQABBHAGIBEM4PJKJVCIPTXNQH3IRH2TDXZ535BRQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 17:26:37.464956223 (505h57m37.336039268s) [noise=true , address=145.79.227.30:4761 , piece_id=QIR3IHTQEWMTQABBHAGIBEM4PJKJVCIPTXNQH3IRH2TDXZ535BRQ, seg_pos={0 0}, upload=443967]: Long tail cancellation due to optimal threshold Dec 26 17:26:37.465142819 (505h57m37.336225864s) [noise=true , address=45.91.116.7:4011 , piece_id=R7YBESBQ4ET2UCWSOPHWDJUXRWBAROX2JXBBNBO2UGCQBGJKWHOQ, seg_pos={0 0}, upload=443967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 17:26:37.465150474 (505h57m37.336233509s) [noise=true , address=45.91.116.7:4011 , piece_id=R7YBESBQ4ET2UCWSOPHWDJUXRWBAROX2JXBBNBO2UGCQBGJKWHOQ, seg_pos={0 0}, upload=443967]: Long tail cancellation due to optimal threshold Dec 26 17:26:37.465474370 (505h57m37.336557415s) [noise=true , address=45.91.116.9:4264 , piece_id=5CLOEFGQUXXEDLROR7HZB4BSIPXEFKBVFLQHRK2WKYMYBXCVK7QA, seg_pos={0 0}, upload=443967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:46928->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:26:37.465483387 (505h57m37.336566432s) [noise=true , address=45.91.116.9:4264 , piece_id=5CLOEFGQUXXEDLROR7HZB4BSIPXEFKBVFLQHRK2WKYMYBXCVK7QA, seg_pos={0 0}, upload=443967]: Long tail cancellation due to optimal threshold Dec 26 17:26:37.465537150 (505h57m37.336620185s) [seg_pos={0 0}, upload=443967]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:26:37.465624507 (505h57m37.336707542s) [upload=443967]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:26:37.465630849 (505h57m37.336713884s) [upload=443967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:26:37.770939229 (505h57m37.642022264s) [upload=443967]: Done uploading segments... Dec 26 17:26:37.770964638 (505h57m37.642047673s) [upload=443967]: Upload finished. err: <nil> Duration: 30.004277147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......X....X..............................................] duration: 40.273414705s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:26:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 40.273450073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.527121655s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:26:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527168986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX...X] duration: 30.002993574s trace: https://jaeger.dev.storj.io/trace/58a6c9a1a1e3e612 FAIL (2025-12-26 17:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:26:02.779381329 (505h57m2.650464374s) [upload=443962]: Starting upload Dec 26 17:26:02.779441785 (505h57m2.65052482s) [upload=443962]: Done starting upload Dec 26 17:26:02.779568727 (505h57m2.650651772s) [upload=443962]: Uploading segments... Dec 26 17:26:02.779684347 (505h57m2.650767392s) [upload=443962]: Got next segment. Inline: false Dec 26 17:26:02.779746074 (505h57m2.65082913s) [upload=443962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:26:02.779753488 (505h57m2.650836523s) [upload=443962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:26:03.030040779 (505h57m2.901123854s) [seg_pos={0 0}, upload=443962]: Begin upload segment... Dec 26 17:26:03.030096154 (505h57m2.9011792s) [seg_pos={0 0}, upload=443962]: Done begin upload segment. Dec 26 17:26:03.030125080 (505h57m2.901208115s) [noise=true , address=77.90.31.17:4773 , piece_id=JN4UCBJIYN4IMNCR6URYEPVTAOUOCX24K5VYMXNE522II2EUZCYA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.030211604 (505h57m2.901294649s) [noise=true , address=145.79.227.38:4978 , piece_id=HO4CQMUGFXIMC3QQUS26D3VAAZKPZYUPQYJWECLGPAQPGPHZLBXQ, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.030370176 (505h57m2.901453251s) [noise=true , address=45.91.116.7:4010 , piece_id=FOCNFHNGGCF6DBD7CNJX7NXD47LLO2IXPYI6JI2HOECGIQ4I3P6Q, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.030794053 (505h57m2.901877088s) [noise=true , address=145.79.227.41:4085 , piece_id=OUPOWLCPZMSXQDF7UJ4ZTKOMJGSR2TOMHTAEA6O56LJH3NHP56LQ, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.031028540 (505h57m2.902111585s) [noise=true , address=45.91.116.6:4973 , piece_id=DBFRU7H7K4YRLV7MIMCINK423OQJ7NQDU2NZATGBE6LG4NPXSTSA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.031166803 (505h57m2.902249838s) [noise=true , address=145.79.227.50:4375 , piece_id=5KJSFRFQ63MX3IONUYTTWSHXYCRRBL6G3BQA57RMARVHEAFHPI6Q, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.031473236 (505h57m2.902556281s) [noise=true , address=145.79.227.19:4077 , piece_id=3KVGD7CQAPUJQRLF5RYKHIQEY76S7P2ZEYCL6XQ6VE2RH3ZBFZGQ, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.031481062 (505h57m2.902564097s) [noise=true , address=145.79.227.45:4230 , piece_id=JIJBCJOWYLU5RFHHYLZYLWLFPXBPYV5A7AVS4XFWCVVKD4IAGBZA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.031851727 (505h57m2.902934772s) [noise=true , address=145.79.227.15:4876 , piece_id=PWRE3RIPJTBNY57ASYNRQNGJZFPD7ZA4S4SSGWSWBYPYKPLRLZMA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.031970764 (505h57m2.903053809s) [noise=true , address=145.79.227.31:4799 , piece_id=TCP37K4K6IK3MG652G5EC3RDZNQUATNIAL4HITCNZKYKZXQ5NRMQ, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.032319889 (505h57m2.903402934s) [noise=true , address=45.91.116.14:4192 , piece_id=L6TUS4GL6JPPLTLIFKVSONOMPYQV6AAZXQCKJ4VOQQHIFOT56ZWA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.032476076 (505h57m2.903559121s) [noise=true , address=145.79.227.21:4475 , piece_id=5SEG7MB54JDC4TEE7YDDHZMZTQBHHLIUKU2H4KQ3MVTYZDJOTUHA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.032689542 (505h57m2.903772587s) [noise=true , address=145.79.227.32:4836 , piece_id=SKV5PEJORTIKJ3XNBHTKGIF3DLJD6P2HQYL7WORLKXC22TFOV5ZA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.033008941 (505h57m2.904091986s) [noise=true , address=45.91.116.9:4261 , piece_id=W6PFILPJDJGY7AFPEVN2VGAEJ6E7XUYWLHJSIZJX4GSL2B7GPBRA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.033025112 (505h57m2.904108157s) [noise=true , address=45.91.116.5:4297 , piece_id=B3WU4IOI2GLXKMFRDI3T3CBO7FKZ5GB6UWJBKS7OS37MS75KZGYA, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:03.033065448 (505h57m2.904148493s) [noise=true , address=145.79.227.14:4839 , piece_id=TSWLNTQ4FRL6BFQEACSN2CV4MDXRS4LSW7QT63EHGPPBWMOFOMUQ, seg_pos={0 0}, upload=443962]: Uploading piece... Dec 26 17:26:04.641991905 (505h57m4.51307494s) [upload=443962]: Next returned nil segment Dec 26 17:26:04.642090392 (505h57m4.513173427s) [upload=443962]: Waiting for error group managing segments... Dec 26 17:26:05.024758009 (505h57m4.895841054s) [noise=true , address=145.79.227.38:4978 , piece_id=HO4CQMUGFXIMC3QQUS26D3VAAZKPZYUPQYJWECLGPAQPGPHZLBXQ, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.025202966 (505h57m4.896286011s) [noise=true , address=145.79.227.21:4475 , piece_id=5SEG7MB54JDC4TEE7YDDHZMZTQBHHLIUKU2H4KQ3MVTYZDJOTUHA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.025755808 (505h57m4.896838843s) [noise=true , address=145.79.227.15:4876 , piece_id=PWRE3RIPJTBNY57ASYNRQNGJZFPD7ZA4S4SSGWSWBYPYKPLRLZMA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.026652836 (505h57m4.897735871s) [noise=true , address=145.79.227.32:4836 , piece_id=SKV5PEJORTIKJ3XNBHTKGIF3DLJD6P2HQYL7WORLKXC22TFOV5ZA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.030583401 (505h57m4.901666446s) [noise=true , address=145.79.227.19:4077 , piece_id=3KVGD7CQAPUJQRLF5RYKHIQEY76S7P2ZEYCL6XQ6VE2RH3ZBFZGQ, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.036443961 (505h57m4.907527006s) [noise=true , address=145.79.227.45:4230 , piece_id=JIJBCJOWYLU5RFHHYLZYLWLFPXBPYV5A7AVS4XFWCVVKD4IAGBZA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.040063704 (505h57m4.911146749s) [noise=true , address=145.79.227.41:4085 , piece_id=OUPOWLCPZMSXQDF7UJ4ZTKOMJGSR2TOMHTAEA6O56LJH3NHP56LQ, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.042708659 (505h57m4.913791704s) [noise=true , address=145.79.227.31:4799 , piece_id=TCP37K4K6IK3MG652G5EC3RDZNQUATNIAL4HITCNZKYKZXQ5NRMQ, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.069940885 (505h57m4.94102395s) [noise=true , address=145.79.227.50:4375 , piece_id=5KJSFRFQ63MX3IONUYTTWSHXYCRRBL6G3BQA57RMARVHEAFHPI6Q, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.267966510 (505h57m5.139049555s) [noise=true , address=45.91.116.14:4192 , piece_id=L6TUS4GL6JPPLTLIFKVSONOMPYQV6AAZXQCKJ4VOQQHIFOT56ZWA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.279382278 (505h57m5.150465323s) [noise=true , address=145.79.227.14:4839 , piece_id=TSWLNTQ4FRL6BFQEACSN2CV4MDXRS4LSW7QT63EHGPPBWMOFOMUQ, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.328812532 (505h57m5.199895587s) [noise=true , address=45.91.116.5:4297 , piece_id=B3WU4IOI2GLXKMFRDI3T3CBO7FKZ5GB6UWJBKS7OS37MS75KZGYA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.366654409 (505h57m5.237737444s) [noise=true , address=45.91.116.6:4973 , piece_id=DBFRU7H7K4YRLV7MIMCINK423OQJ7NQDU2NZATGBE6LG4NPXSTSA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: <nil> Dec 26 17:26:05.366673586 (505h57m5.237756631s) [seg_pos={0 0}, upload=443962]: Segment reached optimal threshold of 13 pieces. Dec 26 17:26:05.366921598 (505h57m5.238004633s) [noise=true , address=45.91.116.9:4261 , piece_id=W6PFILPJDJGY7AFPEVN2VGAEJ6E7XUYWLHJSIZJX4GSL2B7GPBRA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 17:26:05.366929723 (505h57m5.238012758s) [noise=true , address=45.91.116.9:4261 , piece_id=W6PFILPJDJGY7AFPEVN2VGAEJ6E7XUYWLHJSIZJX4GSL2B7GPBRA, seg_pos={0 0}, upload=443962]: Long tail cancellation due to optimal threshold Dec 26 17:26:05.366971232 (505h57m5.238054267s) [noise=true , address=77.90.31.17:4773 , piece_id=JN4UCBJIYN4IMNCR6URYEPVTAOUOCX24K5VYMXNE522II2EUZCYA, seg_pos={0 0}, upload=443962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:35566->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:35566->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35566->77.90.31.17:4773: use of closed network connection Dec 26 17:26:05.366979928 (505h57m5.238062964s) [noise=true , address=77.90.31.17:4773 , piece_id=JN4UCBJIYN4IMNCR6URYEPVTAOUOCX24K5VYMXNE522II2EUZCYA, seg_pos={0 0}, upload=443962]: Long tail cancellation due to optimal threshold Dec 26 17:26:05.367229625 (505h57m5.23831267s) [noise=true , address=45.91.116.7:4010 , piece_id=FOCNFHNGGCF6DBD7CNJX7NXD47LLO2IXPYI6JI2HOECGIQ4I3P6Q, seg_pos={0 0}, upload=443962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 17:26:05.367239283 (505h57m5.238322318s) [noise=true , address=45.91.116.7:4010 , piece_id=FOCNFHNGGCF6DBD7CNJX7NXD47LLO2IXPYI6JI2HOECGIQ4I3P6Q, seg_pos={0 0}, upload=443962]: Long tail cancellation due to optimal threshold Dec 26 17:26:05.367261785 (505h57m5.23834483s) [seg_pos={0 0}, upload=443962]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:26:05.367381093 (505h57m5.238464128s) [upload=443962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:26:05.367388006 (505h57m5.238471051s) [upload=443962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:26:05.667830691 (505h57m5.538913736s) [upload=443962]: Done uploading segments... Dec 26 17:26:05.667850649 (505h57m5.538933684s) [upload=443962]: Upload finished. err: <nil> Duration: 30.003074407s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.519320079s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:26:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519343985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.50679537s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:25:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506824896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X....X.............................................] duration: 25.302246612s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 25.302275919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX...] duration: 6.142102034s trace: https://jaeger.dev.storj.io/trace/1b449e7e7a9cd37e FAIL (2025-12-26 17:25:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ) context deadline exceeded ======LOGS====== 26 17:23:43.339770732 (505h54m43.210853777s) [upload=443930]: Starting upload Dec 26 17:23:43.339964982 (505h54m43.211048047s) [upload=443930]: Done starting upload Dec 26 17:23:43.340280893 (505h54m43.211363928s) [upload=443930]: Uploading segments... Dec 26 17:23:43.340397446 (505h54m43.211480481s) [upload=443930]: Got next segment. Inline: false Dec 26 17:23:43.340449735 (505h54m43.21153278s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:23:43.340457389 (505h54m43.211540434s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:23:43.588670771 (505h54m43.459753816s) [seg_pos={0 0}, upload=443930]: Begin upload segment... Dec 26 17:23:43.588731437 (505h54m43.459814472s) [seg_pos={0 0}, upload=443930]: Done begin upload segment. Dec 26 17:23:43.588766463 (505h54m43.459849508s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589041587 (505h54m43.460124632s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589561057 (505h54m43.460644092s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589927674 (505h54m43.461010709s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590076046 (505h54m43.461159081s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590536855 (505h54m43.46161989s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590572953 (505h54m43.461655988s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590973125 (505h54m43.46205617s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591111378 (505h54m43.462194413s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591382815 (505h54m43.46246585s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591646097 (505h54m43.462729152s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591791183 (505h54m43.462874228s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592243134 (505h54m43.463326209s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592993271 (505h54m43.464076307s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.593697964 (505h54m43.464781009s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.594485974 (505h54m43.465569009s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:45.213346046 (505h54m45.084429081s) [upload=443930]: Next returned nil segment Dec 26 17:23:45.213361315 (505h54m45.08444436s) [upload=443930]: Waiting for error group managing segments... Dec 26 17:23:45.596089386 (505h54m45.467172421s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.599324877 (505h54m45.470407912s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.602939790 (505h54m45.474022835s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.603586763 (505h54m45.474669798s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.615829395 (505h54m45.48691243s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.617160379 (505h54m45.488243414s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.619091246 (505h54m45.490174281s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.774765978 (505h54m45.645849023s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.840220915 (505h54m45.71130396s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.888087351 (505h54m45.759170396s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.946393136 (505h54m45.817476171s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.974051803 (505h54m45.845134838s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.984987386 (505h54m45.856070421s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.985007685 (505h54m45.85609072s) [seg_pos={0 0}, upload=443930]: Segment reached optimal threshold of 13 pieces. Dec 26 17:23:45.985207254 (505h54m45.856290299s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985214358 (505h54m45.856297403s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985372880 (505h54m45.856455935s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection Dec 26 17:23:45.985398228 (505h54m45.856481274s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985458323 (505h54m45.856541358s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985465547 (505h54m45.856548582s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985555988 (505h54m45.856639023s) [seg_pos={0 0}, upload=443930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:23:45.985682930 (505h54m45.856765965s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:23:45.985690104 (505h54m45.856773149s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:23:46.286779621 (505h54m46.157862656s) [upload=443930]: Done uploading segments... Dec 26 17:23:46.286801403 (505h54m46.157884448s) [upload=443930]: Upload finished. err: <nil> Duration: 6.142126801s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X........X..................................................] duration: 3.637290564s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:24:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637334858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX..] duration: 6.066405634s trace: https://jaeger.dev.storj.io/trace/1b449e7e7a9cd37e FAIL (2025-12-26 17:24:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ) context deadline exceeded ======LOGS====== 26 17:23:43.339770732 (505h54m43.210853777s) [upload=443930]: Starting upload Dec 26 17:23:43.339964982 (505h54m43.211048047s) [upload=443930]: Done starting upload Dec 26 17:23:43.340280893 (505h54m43.211363928s) [upload=443930]: Uploading segments... Dec 26 17:23:43.340397446 (505h54m43.211480481s) [upload=443930]: Got next segment. Inline: false Dec 26 17:23:43.340449735 (505h54m43.21153278s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:23:43.340457389 (505h54m43.211540434s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:23:43.588670771 (505h54m43.459753816s) [seg_pos={0 0}, upload=443930]: Begin upload segment... Dec 26 17:23:43.588731437 (505h54m43.459814472s) [seg_pos={0 0}, upload=443930]: Done begin upload segment. Dec 26 17:23:43.588766463 (505h54m43.459849508s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589041587 (505h54m43.460124632s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589561057 (505h54m43.460644092s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589927674 (505h54m43.461010709s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590076046 (505h54m43.461159081s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590536855 (505h54m43.46161989s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590572953 (505h54m43.461655988s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590973125 (505h54m43.46205617s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591111378 (505h54m43.462194413s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591382815 (505h54m43.46246585s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591646097 (505h54m43.462729152s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591791183 (505h54m43.462874228s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592243134 (505h54m43.463326209s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592993271 (505h54m43.464076307s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.593697964 (505h54m43.464781009s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.594485974 (505h54m43.465569009s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:45.213346046 (505h54m45.084429081s) [upload=443930]: Next returned nil segment Dec 26 17:23:45.213361315 (505h54m45.08444436s) [upload=443930]: Waiting for error group managing segments... Dec 26 17:23:45.596089386 (505h54m45.467172421s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.599324877 (505h54m45.470407912s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.602939790 (505h54m45.474022835s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.603586763 (505h54m45.474669798s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.615829395 (505h54m45.48691243s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.617160379 (505h54m45.488243414s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.619091246 (505h54m45.490174281s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.774765978 (505h54m45.645849023s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.840220915 (505h54m45.71130396s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.888087351 (505h54m45.759170396s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.946393136 (505h54m45.817476171s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.974051803 (505h54m45.845134838s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.984987386 (505h54m45.856070421s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.985007685 (505h54m45.85609072s) [seg_pos={0 0}, upload=443930]: Segment reached optimal threshold of 13 pieces. Dec 26 17:23:45.985207254 (505h54m45.856290299s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985214358 (505h54m45.856297403s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985372880 (505h54m45.856455935s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection Dec 26 17:23:45.985398228 (505h54m45.856481274s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985458323 (505h54m45.856541358s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985465547 (505h54m45.856548582s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985555988 (505h54m45.856639023s) [seg_pos={0 0}, upload=443930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:23:45.985682930 (505h54m45.856765965s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:23:45.985690104 (505h54m45.856773149s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:23:46.286779621 (505h54m46.157862656s) [upload=443930]: Done uploading segments... Dec 26 17:23:46.286801403 (505h54m46.157884448s) [upload=443930]: Upload finished. err: <nil> Duration: 6.06643479s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X........X.................................................] duration: 3.561172563s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:24:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.56126007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X....X............................................] duration: 35.645818896s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:24:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 35.645855336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX.] duration: 9.641287894s trace: https://jaeger.dev.storj.io/trace/1b449e7e7a9cd37e FAIL (2025-12-26 17:24:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ) context deadline exceeded ======LOGS====== 26 17:23:43.339770732 (505h54m43.210853777s) [upload=443930]: Starting upload Dec 26 17:23:43.339964982 (505h54m43.211048047s) [upload=443930]: Done starting upload Dec 26 17:23:43.340280893 (505h54m43.211363928s) [upload=443930]: Uploading segments... Dec 26 17:23:43.340397446 (505h54m43.211480481s) [upload=443930]: Got next segment. Inline: false Dec 26 17:23:43.340449735 (505h54m43.21153278s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:23:43.340457389 (505h54m43.211540434s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:23:43.588670771 (505h54m43.459753816s) [seg_pos={0 0}, upload=443930]: Begin upload segment... Dec 26 17:23:43.588731437 (505h54m43.459814472s) [seg_pos={0 0}, upload=443930]: Done begin upload segment. Dec 26 17:23:43.588766463 (505h54m43.459849508s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589041587 (505h54m43.460124632s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589561057 (505h54m43.460644092s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589927674 (505h54m43.461010709s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590076046 (505h54m43.461159081s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590536855 (505h54m43.46161989s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590572953 (505h54m43.461655988s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590973125 (505h54m43.46205617s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591111378 (505h54m43.462194413s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591382815 (505h54m43.46246585s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591646097 (505h54m43.462729152s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591791183 (505h54m43.462874228s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592243134 (505h54m43.463326209s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592993271 (505h54m43.464076307s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.593697964 (505h54m43.464781009s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.594485974 (505h54m43.465569009s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:45.213346046 (505h54m45.084429081s) [upload=443930]: Next returned nil segment Dec 26 17:23:45.213361315 (505h54m45.08444436s) [upload=443930]: Waiting for error group managing segments... Dec 26 17:23:45.596089386 (505h54m45.467172421s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.599324877 (505h54m45.470407912s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.602939790 (505h54m45.474022835s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.603586763 (505h54m45.474669798s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.615829395 (505h54m45.48691243s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.617160379 (505h54m45.488243414s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.619091246 (505h54m45.490174281s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.774765978 (505h54m45.645849023s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.840220915 (505h54m45.71130396s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.888087351 (505h54m45.759170396s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.946393136 (505h54m45.817476171s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.974051803 (505h54m45.845134838s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.984987386 (505h54m45.856070421s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.985007685 (505h54m45.85609072s) [seg_pos={0 0}, upload=443930]: Segment reached optimal threshold of 13 pieces. Dec 26 17:23:45.985207254 (505h54m45.856290299s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985214358 (505h54m45.856297403s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985372880 (505h54m45.856455935s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection Dec 26 17:23:45.985398228 (505h54m45.856481274s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985458323 (505h54m45.856541358s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985465547 (505h54m45.856548582s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985555988 (505h54m45.856639023s) [seg_pos={0 0}, upload=443930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:23:45.985682930 (505h54m45.856765965s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:23:45.985690104 (505h54m45.856773149s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:23:46.286779621 (505h54m46.157862656s) [upload=443930]: Done uploading segments... Dec 26 17:23:46.286801403 (505h54m46.157884448s) [upload=443930]: Upload finished. err: <nil> Duration: 9.641326647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...XX] duration: 30.002902209s trace: https://jaeger.dev.storj.io/trace/1b449e7e7a9cd37e FAIL (2025-12-26 17:24:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ) context deadline exceeded ======LOGS====== 26 17:23:43.339770732 (505h54m43.210853777s) [upload=443930]: Starting upload Dec 26 17:23:43.339964982 (505h54m43.211048047s) [upload=443930]: Done starting upload Dec 26 17:23:43.340280893 (505h54m43.211363928s) [upload=443930]: Uploading segments... Dec 26 17:23:43.340397446 (505h54m43.211480481s) [upload=443930]: Got next segment. Inline: false Dec 26 17:23:43.340449735 (505h54m43.21153278s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:23:43.340457389 (505h54m43.211540434s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:23:43.588670771 (505h54m43.459753816s) [seg_pos={0 0}, upload=443930]: Begin upload segment... Dec 26 17:23:43.588731437 (505h54m43.459814472s) [seg_pos={0 0}, upload=443930]: Done begin upload segment. Dec 26 17:23:43.588766463 (505h54m43.459849508s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589041587 (505h54m43.460124632s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589561057 (505h54m43.460644092s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.589927674 (505h54m43.461010709s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590076046 (505h54m43.461159081s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590536855 (505h54m43.46161989s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590572953 (505h54m43.461655988s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.590973125 (505h54m43.46205617s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591111378 (505h54m43.462194413s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591382815 (505h54m43.46246585s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591646097 (505h54m43.462729152s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.591791183 (505h54m43.462874228s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592243134 (505h54m43.463326209s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.592993271 (505h54m43.464076307s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.593697964 (505h54m43.464781009s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:43.594485974 (505h54m43.465569009s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Uploading piece... Dec 26 17:23:45.213346046 (505h54m45.084429081s) [upload=443930]: Next returned nil segment Dec 26 17:23:45.213361315 (505h54m45.08444436s) [upload=443930]: Waiting for error group managing segments... Dec 26 17:23:45.596089386 (505h54m45.467172421s) [noise=true , address=145.79.227.40:4049 , piece_id=DTAQBJ3ZL5OLE6HOYOGXYYVQQUTYC3HOQPUYGKIQQQ7WAS3QBXKA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.599324877 (505h54m45.470407912s) [noise=true , address=145.79.227.14:4839 , piece_id=UX6ST3FML2KTYEIRSIBZRTJOIVWBIX4QOTUH73HGJ2DRZLU6YMVQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.602939790 (505h54m45.474022835s) [noise=true , address=145.79.227.43:4159 , piece_id=XMRJFLRO5NY4KXUKZFVHFKL5MY5FA3H4VTESQJEJZWSLQSRTH73Q, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.603586763 (505h54m45.474669798s) [noise=true , address=145.79.227.11:4731 , piece_id=IISUQXLEGDVTM2ELPZ3FFWWICRRYXZNPMPUCY3KOOLV2MMKPICPA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.615829395 (505h54m45.48691243s) [noise=true , address=145.79.227.37:4566 , piece_id=M7UNKG37X45XKBCYAZ77VZGZ25ORHKCSNCN3DIJFJQPZPTO572LA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.617160379 (505h54m45.488243414s) [noise=true , address=145.79.227.12:4767 , piece_id=GDQBTOJY3CCYBX6WH3XEXH4NFF4NN2EBHMMG6Y6IEUH2SRKFOFIQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.619091246 (505h54m45.490174281s) [noise=true , address=145.79.227.16:4911 , piece_id=G5SN46IFLA3H7JD2DKEK2BMQA2JJHXPBCF7EWZECMSV6G3PFO4XQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.774765978 (505h54m45.645849023s) [noise=true , address=77.90.31.17:4773 , piece_id=P6ALNYHZRMWE25NM5GMP342B2M6Q4M6VKEM5XQDYIHTXA3WUT7PA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.840220915 (505h54m45.71130396s) [noise=true , address=45.91.116.10:4082 , piece_id=T37KDXWG6JN3DP6LGNWUICSU2FRDNIGRQR7T2XCJFGQOQPOB663A, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.888087351 (505h54m45.759170396s) [noise=true , address=77.90.31.14:4663 , piece_id=FYRHSOBX7ARV52PHPCDOODSFQAANA665TNRRWL4J26EJEDEAEHLA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.946393136 (505h54m45.817476171s) [noise=true , address=45.91.116.9:4264 , piece_id=ZP653KXUQWJAB6C452YLYEFCUK72V5IEYAZITCMLB6NLY4ZMY6YQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.974051803 (505h54m45.845134838s) [noise=true , address=45.91.116.6:4974 , piece_id=WRHW6ZXB22CKH5O4CPPG6G2UHCODFSHFVPGA36QNFJA7NF4PAIDQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.984987386 (505h54m45.856070421s) [noise=true , address=45.91.116.14:4191 , piece_id=D72AL6YUXYZZGHP7DLWHLJS3OCF3JYAV2G5HFJJPE7JNPBQJB6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: <nil> Dec 26 17:23:45.985007685 (505h54m45.85609072s) [seg_pos={0 0}, upload=443930]: Segment reached optimal threshold of 13 pieces. Dec 26 17:23:45.985207254 (505h54m45.856290299s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985214358 (505h54m45.856297403s) [noise=true , address=45.91.116.11:4119 , piece_id=BUY7L5UEYZZGLKJUFRTZHXZ5EFE33V6S7TRDNLPFQTMU3L62KNHQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985372880 (505h54m45.856455935s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43400->145.79.227.6:4257: use of closed network connection Dec 26 17:23:45.985398228 (505h54m45.856481274s) [noise=true , address=145.79.227.6:4257 , piece_id=2PPJJ2II3U3WWRJQWWHUSUPJQ65ECMTMS32BBQPIPFS47AXZS6RQ, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985458323 (505h54m45.856541358s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:23:45.985465547 (505h54m45.856548582s) [noise=true , address=45.91.116.12:4228 , piece_id=3XMNGWSHPZMOQQ54M6DSD7LV4DUHXUHRS2JZ5URHRHKRGQTL7OVA, seg_pos={0 0}, upload=443930]: Long tail cancellation due to optimal threshold Dec 26 17:23:45.985555988 (505h54m45.856639023s) [seg_pos={0 0}, upload=443930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:23:45.985682930 (505h54m45.856765965s) [upload=443930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:23:45.985690104 (505h54m45.856773149s) [upload=443930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:23:46.286779621 (505h54m46.157862656s) [upload=443930]: Done uploading segments... Dec 26 17:23:46.286801403 (505h54m46.157884448s) [upload=443930]: Upload finished. err: <nil> Duration: 30.002913691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X........X................................................] duration: 3.625726055s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:23:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.625747045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...X] duration: 30.00293809s trace: https://jaeger.dev.storj.io/trace/6c67148cfa6e1d53 FAIL (2025-12-26 17:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load Dec 26 17:23:13.114377650 (505h54m12.985460685s) [upload=443921]: Done starting upload Dec 26 17:23:13.114480185 (505h54m12.98556322s) [upload=443921]: Uploading segments... Dec 26 17:23:13.114626163 (505h54m12.985709198s) [upload=443921]: Got next segment. Inline: false Dec 26 17:23:13.114670758 (505h54m12.985753803s) [upload=443921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:23:13.114675046 (505h54m12.985758081s) [upload=443921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:23:13.374381682 (505h54m13.245464727s) [seg_pos={0 0}, upload=443921]: Begin upload segment... Dec 26 17:23:13.374443731 (505h54m13.245526766s) [seg_pos={0 0}, upload=443921]: Done begin upload segment. Dec 26 17:23:13.374475762 (505h54m13.245558807s) [noise=true , address=77.90.31.12:4590 , piece_id=IDGSIBVKWMLKXYO442EEAFMJIVXMYCAHM6CVDUSLIHFMEWT2NOFQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.379623023 (505h54m13.250706068s) [noise=true , address=145.79.227.38:4977 , piece_id=7CCMW3P7IASD4SEPDFOK3XVFV6N3ZZINLHVVDIKQ7B5WDKUL4EKA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.381208362 (505h54m13.252291397s) [noise=true , address=145.79.227.37:4566 , piece_id=SGDS3FYYRO76YVM5OIXNXYM26CPL6EJ2T4H2YKVITO7KPZ24TDEQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.382711124 (505h54m13.253794169s) [noise=true , address=45.91.116.9:4261 , piece_id=3NAV7IY7XJEHVYDIDZPNFJYAMXKAA6HZVRHRXDKUHNKFGCIF6H7Q, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.384123202 (505h54m13.255206247s) [noise=true , address=77.90.31.13:4629 , piece_id=XIQVFRDLU35RNRNGNFPMLBNDG4HY6TD5POAPA5HM7SWGZ6AHDYRQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.384328703 (505h54m13.255411738s) [noise=true , address=145.79.227.32:4836 , piece_id=V4VETZZO4PDMFFAWKTE4QANOO75NQUD4A7FQTY5V3JJZWKGXYMXA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.385484504 (505h54m13.256567549s) [noise=true , address=145.79.227.48:4301 , piece_id=IOYXVXL6G4KQGG7SYDDSIY6SUYSTWXYKW7DHIBX7M5VG3UII2NWQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.386621620 (505h54m13.257704655s) [noise=true , address=145.79.227.4:4187 , piece_id=FFW6MQBHX4EWWI27FZQC3MYJF45TDEBNG5A4TKHOEZ6AOBOKLMCQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.386872036 (505h54m13.257955081s) [noise=true , address=45.91.116.5:4299 , piece_id=SI6VNWWCXSNK7WMUWAXXXKK7PDGQQEKREEVHZG4NKUMZPCUDY7RQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.388277283 (505h54m13.259360328s) [noise=true , address=145.79.227.46:4603 , piece_id=5ROIFH7BBZIBXT33JGOIIVHJSFE5RYVWZCUZJ3VCP4FTJ52XH2MQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.388345703 (505h54m13.259428738s) [noise=true , address=145.79.227.11:4732 , piece_id=FZ2FKRZZJXQ2I67VUUN22GP6EYEWOVLYRKYLL26XCGKAKBJ4CMWA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.389425369 (505h54m13.260508404s) [noise=true , address=145.79.227.13:4803 , piece_id=RZGV43VZDAHWF6P6GTUA5LLRSIE4PA64Q3Y7CHKYHFVCRRLPGNHA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.390610886 (505h54m13.261693921s) [noise=true , address=145.79.227.22:4509 , piece_id=4SMQKWLS6TXFL4TCQ67JJWPMBNVTBGE6P6MB2QJHGCDK2XWIQXCA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.390736675 (505h54m13.26181971s) [noise=true , address=45.91.116.11:4119 , piece_id=E2AVJJZKRASQLSMB2PWMIBINJ4WSTFGG2FAZII3D3HR7FZBU6QEQ, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.392010631 (505h54m13.263093667s) [noise=true , address=45.91.116.7:4011 , piece_id=BC6AZTEOCWQMIQCKAHYR2RT6F6WOEYZK4KSQBDS2ZNMABPXQGYMA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:13.392151730 (505h54m13.263234765s) [noise=true , address=45.91.116.10:4081 , piece_id=XIYUGYXJCHMOG2R3GEZH4MBP4YZ66HVEHMILIXXLVXYYTAYTBGTA, seg_pos={0 0}, upload=443921]: Uploading piece... Dec 26 17:23:15.088087528 (505h54m14.959170573s) [upload=443921]: Next returned nil segment Dec 26 17:23:15.088103638 (505h54m14.959186673s) [upload=443921]: Waiting for error group managing segments... Dec 26 17:23:15.383481589 (505h54m15.254564625s) [noise=true , address=145.79.227.48:4301 , piece_id=IOYXVXL6G4KQGG7SYDDSIY6SUYSTWXYKW7DHIBX7M5VG3UII2NWQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.383890188 (505h54m15.254973223s) [noise=true , address=145.79.227.37:4566 , piece_id=SGDS3FYYRO76YVM5OIXNXYM26CPL6EJ2T4H2YKVITO7KPZ24TDEQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.384223903 (505h54m15.255306938s) [noise=true , address=145.79.227.11:4732 , piece_id=FZ2FKRZZJXQ2I67VUUN22GP6EYEWOVLYRKYLL26XCGKAKBJ4CMWA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.384581374 (505h54m15.255664409s) [noise=true , address=145.79.227.38:4977 , piece_id=7CCMW3P7IASD4SEPDFOK3XVFV6N3ZZINLHVVDIKQ7B5WDKUL4EKA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.388282020 (505h54m15.259365065s) [noise=true , address=145.79.227.32:4836 , piece_id=V4VETZZO4PDMFFAWKTE4QANOO75NQUD4A7FQTY5V3JJZWKGXYMXA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.413007071 (505h54m15.284090106s) [noise=true , address=145.79.227.4:4187 , piece_id=FFW6MQBHX4EWWI27FZQC3MYJF45TDEBNG5A4TKHOEZ6AOBOKLMCQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.413510500 (505h54m15.284593535s) [noise=true , address=145.79.227.22:4509 , piece_id=4SMQKWLS6TXFL4TCQ67JJWPMBNVTBGE6P6MB2QJHGCDK2XWIQXCA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.414550020 (505h54m15.285633055s) [noise=true , address=145.79.227.46:4603 , piece_id=5ROIFH7BBZIBXT33JGOIIVHJSFE5RYVWZCUZJ3VCP4FTJ52XH2MQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.433930212 (505h54m15.305013247s) [noise=true , address=145.79.227.13:4803 , piece_id=RZGV43VZDAHWF6P6GTUA5LLRSIE4PA64Q3Y7CHKYHFVCRRLPGNHA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.642950343 (505h54m15.514033388s) [noise=true , address=45.91.116.11:4119 , piece_id=E2AVJJZKRASQLSMB2PWMIBINJ4WSTFGG2FAZII3D3HR7FZBU6QEQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.667470614 (505h54m15.53855366s) [noise=true , address=45.91.116.7:4011 , piece_id=BC6AZTEOCWQMIQCKAHYR2RT6F6WOEYZK4KSQBDS2ZNMABPXQGYMA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.807194383 (505h54m15.678277428s) [noise=true , address=45.91.116.9:4261 , piece_id=3NAV7IY7XJEHVYDIDZPNFJYAMXKAA6HZVRHRXDKUHNKFGCIF6H7Q, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.872917521 (505h54m15.744000566s) [noise=true , address=77.90.31.12:4590 , piece_id=IDGSIBVKWMLKXYO442EEAFMJIVXMYCAHM6CVDUSLIHFMEWT2NOFQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: <nil> Dec 26 17:23:15.872941176 (505h54m15.744024221s) [seg_pos={0 0}, upload=443921]: Segment reached optimal threshold of 13 pieces. Dec 26 17:23:15.873371355 (505h54m15.74445443s) [noise=true , address=77.90.31.13:4629 , piece_id=XIQVFRDLU35RNRNGNFPMLBNDG4HY6TD5POAPA5HM7SWGZ6AHDYRQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:35980->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:23:15.873382837 (505h54m15.744465882s) [noise=true , address=77.90.31.13:4629 , piece_id=XIQVFRDLU35RNRNGNFPMLBNDG4HY6TD5POAPA5HM7SWGZ6AHDYRQ, seg_pos={0 0}, upload=443921]: Long tail cancellation due to optimal threshold Dec 26 17:23:15.873454142 (505h54m15.744537208s) [noise=true , address=45.91.116.10:4081 , piece_id=XIYUGYXJCHMOG2R3GEZH4MBP4YZ66HVEHMILIXXLVXYYTAYTBGTA, seg_pos={0 0}, upload=443921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:59664->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:23:15.873462750 (505h54m15.744545805s) [noise=true , address=45.91.116.10:4081 , piece_id=XIYUGYXJCHMOG2R3GEZH4MBP4YZ66HVEHMILIXXLVXYYTAYTBGTA, seg_pos={0 0}, upload=443921]: Long tail cancellation due to optimal threshold Dec 26 17:23:15.873508677 (505h54m15.744591722s) [noise=true , address=45.91.116.5:4299 , piece_id=SI6VNWWCXSNK7WMUWAXXXKK7PDGQQEKREEVHZG4NKUMZPCUDY7RQ, seg_pos={0 0}, upload=443921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:32852->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:23:15.873517483 (505h54m15.744600518s) [noise=true , address=45.91.116.5:4299 , piece_id=SI6VNWWCXSNK7WMUWAXXXKK7PDGQQEKREEVHZG4NKUMZPCUDY7RQ, seg_pos={0 0}, upload=443921]: Long tail cancellation due to optimal threshold Dec 26 17:23:15.873537352 (505h54m15.744620397s) [seg_pos={0 0}, upload=443921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:23:15.873619748 (505h54m15.744702783s) [upload=443921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:23:15.873625990 (505h54m15.744709025s) [upload=443921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:23:16.166407798 (505h54m16.037490843s) [upload=443921]: Done uploading segments... Dec 26 17:23:16.166431703 (505h54m16.037514738s) [upload=443921]: Upload finished. err: <nil> Duration: 30.002963037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......X....X...........................................] duration: 38.883288393s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 38.883302068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X....X..........................................] duration: 47.187946842s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:22:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 47.187980628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X........X...............................................] duration: 3.485338743s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:22:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485367026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX...] duration: 5.981366435s trace: https://jaeger.dev.storj.io/trace/215b8c1fafd51df8 FAIL (2025-12-26 17:22:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ) context deadline exceeded ======LOGS====== Dec 26 17:20:34.895994575 (505h51m34.76707762s) [upload=443884]: Starting upload Dec 26 17:20:34.896136114 (505h51m34.767219169s) [upload=443884]: Done starting upload Dec 26 17:20:34.896453660 (505h51m34.767536705s) [upload=443884]: Uploading segments... Dec 26 17:20:34.896502532 (505h51m34.767585577s) [upload=443884]: Got next segment. Inline: false Dec 26 17:20:34.896535936 (505h51m34.767618971s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:20:34.896541266 (505h51m34.767624321s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:20:35.150639133 (505h51m35.021722188s) [seg_pos={0 0}, upload=443884]: Begin upload segment... Dec 26 17:20:35.150712132 (505h51m35.021795167s) [seg_pos={0 0}, upload=443884]: Done begin upload segment. Dec 26 17:20:35.150747689 (505h51m35.021830734s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.150914146 (505h51m35.021997182s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.151448574 (505h51m35.022531619s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152105515 (505h51m35.02318855s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152823402 (505h51m35.023906447s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153257719 (505h51m35.024340754s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153485934 (505h51m35.024568969s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153854395 (505h51m35.02493743s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153968062 (505h51m35.025051107s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154557083 (505h51m35.025640128s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154666632 (505h51m35.025749677s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155040563 (505h51m35.026123608s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155168227 (505h51m35.026251262s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155655455 (505h51m35.0267385s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156139466 (505h51m35.027222501s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156868845 (505h51m35.02795189s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:36.869605965 (505h51m36.74068901s) [upload=443884]: Next returned nil segment Dec 26 17:20:36.869621194 (505h51m36.74070423s) [upload=443884]: Waiting for error group managing segments... Dec 26 17:20:36.986631823 (505h51m36.857714868s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:20:37.149179592 (505h51m37.020262627s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.156921996 (505h51m37.028005081s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.163105449 (505h51m37.034188484s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.166262321 (505h51m37.037345366s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.170864023 (505h51m37.041947058s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.173630441 (505h51m37.044713476s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.178820284 (505h51m37.049903319s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.184576574 (505h51m37.055659609s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.197239156 (505h51m37.068330086s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.409764220 (505h51m37.280847256s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.456956613 (505h51m37.328039648s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.461704615 (505h51m37.33278765s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548235866 (505h51m37.419318901s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548255252 (505h51m37.419338287s) [seg_pos={0 0}, upload=443884]: Segment reached optimal threshold of 13 pieces. Dec 26 17:20:37.548427720 (505h51m37.419510765s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548434683 (505h51m37.419517719s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548742390 (505h51m37.419825425s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548749533 (505h51m37.419832578s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548766586 (505h51m37.419849621s) [seg_pos={0 0}, upload=443884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:20:37.548833213 (505h51m37.419916248s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:20:37.548837932 (505h51m37.419920977s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:20:37.848539526 (505h51m37.719622561s) [upload=443884]: Done uploading segments... Dec 26 17:20:37.848559173 (505h51m37.719642208s) [upload=443884]: Upload finished. err: <nil> Duration: 5.981394979s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X...X........X..............................................] duration: 3.452442873s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:22:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452473432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX..] duration: 6.236080235s trace: https://jaeger.dev.storj.io/trace/215b8c1fafd51df8 FAIL (2025-12-26 17:21:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ) context deadline exceeded ======LOGS====== Dec 26 17:20:34.895994575 (505h51m34.76707762s) [upload=443884]: Starting upload Dec 26 17:20:34.896136114 (505h51m34.767219169s) [upload=443884]: Done starting upload Dec 26 17:20:34.896453660 (505h51m34.767536705s) [upload=443884]: Uploading segments... Dec 26 17:20:34.896502532 (505h51m34.767585577s) [upload=443884]: Got next segment. Inline: false Dec 26 17:20:34.896535936 (505h51m34.767618971s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:20:34.896541266 (505h51m34.767624321s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:20:35.150639133 (505h51m35.021722188s) [seg_pos={0 0}, upload=443884]: Begin upload segment... Dec 26 17:20:35.150712132 (505h51m35.021795167s) [seg_pos={0 0}, upload=443884]: Done begin upload segment. Dec 26 17:20:35.150747689 (505h51m35.021830734s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.150914146 (505h51m35.021997182s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.151448574 (505h51m35.022531619s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152105515 (505h51m35.02318855s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152823402 (505h51m35.023906447s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153257719 (505h51m35.024340754s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153485934 (505h51m35.024568969s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153854395 (505h51m35.02493743s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153968062 (505h51m35.025051107s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154557083 (505h51m35.025640128s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154666632 (505h51m35.025749677s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155040563 (505h51m35.026123608s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155168227 (505h51m35.026251262s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155655455 (505h51m35.0267385s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156139466 (505h51m35.027222501s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156868845 (505h51m35.02795189s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:36.869605965 (505h51m36.74068901s) [upload=443884]: Next returned nil segment Dec 26 17:20:36.869621194 (505h51m36.74070423s) [upload=443884]: Waiting for error group managing segments... Dec 26 17:20:36.986631823 (505h51m36.857714868s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:20:37.149179592 (505h51m37.020262627s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.156921996 (505h51m37.028005081s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.163105449 (505h51m37.034188484s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.166262321 (505h51m37.037345366s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.170864023 (505h51m37.041947058s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.173630441 (505h51m37.044713476s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.178820284 (505h51m37.049903319s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.184576574 (505h51m37.055659609s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.197239156 (505h51m37.068330086s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.409764220 (505h51m37.280847256s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.456956613 (505h51m37.328039648s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.461704615 (505h51m37.33278765s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548235866 (505h51m37.419318901s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548255252 (505h51m37.419338287s) [seg_pos={0 0}, upload=443884]: Segment reached optimal threshold of 13 pieces. Dec 26 17:20:37.548427720 (505h51m37.419510765s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548434683 (505h51m37.419517719s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548742390 (505h51m37.419825425s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548749533 (505h51m37.419832578s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548766586 (505h51m37.419849621s) [seg_pos={0 0}, upload=443884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:20:37.548833213 (505h51m37.419916248s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:20:37.548837932 (505h51m37.419920977s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:20:37.848539526 (505h51m37.719622561s) [upload=443884]: Done uploading segments... Dec 26 17:20:37.848559173 (505h51m37.719642208s) [upload=443884]: Upload finished. err: <nil> Duration: 6.236103068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XX...X........X.............................................] duration: 3.516269522s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:21:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516296992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X....X.........................................] duration: 40.637108549s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 40.637126122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX.] duration: 5.882146032s trace: https://jaeger.dev.storj.io/trace/215b8c1fafd51df8 FAIL (2025-12-26 17:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ) context deadline exceeded ======LOGS====== Dec 26 17:20:34.895994575 (505h51m34.76707762s) [upload=443884]: Starting upload Dec 26 17:20:34.896136114 (505h51m34.767219169s) [upload=443884]: Done starting upload Dec 26 17:20:34.896453660 (505h51m34.767536705s) [upload=443884]: Uploading segments... Dec 26 17:20:34.896502532 (505h51m34.767585577s) [upload=443884]: Got next segment. Inline: false Dec 26 17:20:34.896535936 (505h51m34.767618971s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:20:34.896541266 (505h51m34.767624321s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:20:35.150639133 (505h51m35.021722188s) [seg_pos={0 0}, upload=443884]: Begin upload segment... Dec 26 17:20:35.150712132 (505h51m35.021795167s) [seg_pos={0 0}, upload=443884]: Done begin upload segment. Dec 26 17:20:35.150747689 (505h51m35.021830734s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.150914146 (505h51m35.021997182s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.151448574 (505h51m35.022531619s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152105515 (505h51m35.02318855s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152823402 (505h51m35.023906447s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153257719 (505h51m35.024340754s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153485934 (505h51m35.024568969s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153854395 (505h51m35.02493743s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153968062 (505h51m35.025051107s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154557083 (505h51m35.025640128s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154666632 (505h51m35.025749677s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155040563 (505h51m35.026123608s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155168227 (505h51m35.026251262s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155655455 (505h51m35.0267385s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156139466 (505h51m35.027222501s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156868845 (505h51m35.02795189s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:36.869605965 (505h51m36.74068901s) [upload=443884]: Next returned nil segment Dec 26 17:20:36.869621194 (505h51m36.74070423s) [upload=443884]: Waiting for error group managing segments... Dec 26 17:20:36.986631823 (505h51m36.857714868s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:20:37.149179592 (505h51m37.020262627s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.156921996 (505h51m37.028005081s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.163105449 (505h51m37.034188484s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.166262321 (505h51m37.037345366s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.170864023 (505h51m37.041947058s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.173630441 (505h51m37.044713476s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.178820284 (505h51m37.049903319s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.184576574 (505h51m37.055659609s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.197239156 (505h51m37.068330086s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.409764220 (505h51m37.280847256s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.456956613 (505h51m37.328039648s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.461704615 (505h51m37.33278765s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548235866 (505h51m37.419318901s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548255252 (505h51m37.419338287s) [seg_pos={0 0}, upload=443884]: Segment reached optimal threshold of 13 pieces. Dec 26 17:20:37.548427720 (505h51m37.419510765s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548434683 (505h51m37.419517719s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548742390 (505h51m37.419825425s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548749533 (505h51m37.419832578s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548766586 (505h51m37.419849621s) [seg_pos={0 0}, upload=443884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:20:37.548833213 (505h51m37.419916248s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:20:37.548837932 (505h51m37.419920977s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:20:37.848539526 (505h51m37.719622561s) [upload=443884]: Done uploading segments... Dec 26 17:20:37.848559173 (505h51m37.719642208s) [upload=443884]: Upload finished. err: <nil> Duration: 5.882173254s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [XXX...X........X............................................] duration: 3.422613623s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:21:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422643039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.XX] duration: 30.002920956s trace: https://jaeger.dev.storj.io/trace/215b8c1fafd51df8 FAIL (2025-12-26 17:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ) context deadline exceeded ======LOGS====== Dec 26 17:20:34.895994575 (505h51m34.76707762s) [upload=443884]: Starting upload Dec 26 17:20:34.896136114 (505h51m34.767219169s) [upload=443884]: Done starting upload Dec 26 17:20:34.896453660 (505h51m34.767536705s) [upload=443884]: Uploading segments... Dec 26 17:20:34.896502532 (505h51m34.767585577s) [upload=443884]: Got next segment. Inline: false Dec 26 17:20:34.896535936 (505h51m34.767618971s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:20:34.896541266 (505h51m34.767624321s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:20:35.150639133 (505h51m35.021722188s) [seg_pos={0 0}, upload=443884]: Begin upload segment... Dec 26 17:20:35.150712132 (505h51m35.021795167s) [seg_pos={0 0}, upload=443884]: Done begin upload segment. Dec 26 17:20:35.150747689 (505h51m35.021830734s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.150914146 (505h51m35.021997182s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.151448574 (505h51m35.022531619s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152105515 (505h51m35.02318855s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.152823402 (505h51m35.023906447s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153257719 (505h51m35.024340754s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153485934 (505h51m35.024568969s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153854395 (505h51m35.02493743s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.153968062 (505h51m35.025051107s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154557083 (505h51m35.025640128s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.154666632 (505h51m35.025749677s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155040563 (505h51m35.026123608s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155168227 (505h51m35.026251262s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.155655455 (505h51m35.0267385s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156139466 (505h51m35.027222501s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:35.156868845 (505h51m35.02795189s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Uploading piece... Dec 26 17:20:36.869605965 (505h51m36.74068901s) [upload=443884]: Next returned nil segment Dec 26 17:20:36.869621194 (505h51m36.74070423s) [upload=443884]: Waiting for error group managing segments... Dec 26 17:20:36.986631823 (505h51m36.857714868s) [noise=true , address=77.90.31.12:4589 , piece_id=5YR654AKXBRGWQJFMVIG6YQO22Y4NRNQHI23HNWH7O5KUEX2ESUQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:20:37.149179592 (505h51m37.020262627s) [noise=true , address=145.79.227.35:4481 , piece_id=KXOUGRIMDV4K7XBXLYOERYRVKMQY7UM7NZCWZUP43CWGBKKZ243Q, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.156921996 (505h51m37.028005081s) [noise=true , address=145.79.227.8:4330 , piece_id=3OXAXSSQ4DKYD7NXX5KNGFSDINTMUDBHVPBLTE3TW7I2B3DJ4AOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.163105449 (505h51m37.034188484s) [noise=true , address=145.79.227.31:4799 , piece_id=67P2C65ODPS2TVTPMVEFYP4MENLATXZO46G46KI2JZKPOX2SOEJQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.166262321 (505h51m37.037345366s) [noise=true , address=145.79.227.42:4121 , piece_id=I3TVCYQJX4JKBHGT4GXFRI5TFWUKJXF4QPXWTWMCC3QYHK73YWAA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.170864023 (505h51m37.041947058s) [noise=true , address=145.79.227.4:4187 , piece_id=NENKTT6ZOAUVR7ASUW57IJLN5G2DLVYUOQW45CO44MWPXT6MELZA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.173630441 (505h51m37.044713476s) [noise=true , address=145.79.227.15:4874 , piece_id=TVCLODXTM3TAT72RY7ZGWWBLPR74ZI2KB2SJU3MAWSG26CTY7DOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.178820284 (505h51m37.049903319s) [noise=true , address=145.79.227.3:4151 , piece_id=6A7FNASMNYBY4OAAD7NCSEA6PRE7YEFHTG4BAL7CXWO3KRAVO2FQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.184576574 (505h51m37.055659609s) [noise=true , address=145.79.227.46:4603 , piece_id=AUXIGWWBMYJJOQYYGIKAV6XKBWA7I5E27HTBQ2CT2K4ACWY4HQYA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.197239156 (505h51m37.068330086s) [noise=true , address=145.79.227.5:4223 , piece_id=KWFOLVPBGAF3TXM6SEI2P5JYXXYHWWI76BLETVFYBUWEUCARPQOA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.409764220 (505h51m37.280847256s) [noise=true , address=45.91.116.6:4974 , piece_id=ZBWJ7ORBG4QTF336EPAH6TJYF5RYI3UYOAD3B26PS4KTHLAGX2UQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.456956613 (505h51m37.328039648s) [noise=true , address=77.90.31.10:4517 , piece_id=EPXMZXMMCRKY7HGMPG7KWQW5TJWU2GZI7WRGQTMGQJ4GPX6334DA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.461704615 (505h51m37.33278765s) [noise=true , address=45.91.116.14:4189 , piece_id=K42MVJR7LIIC33BASGIBQAQBZWZVB4GW2FUC35DN3EFDGN32YSBA, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548235866 (505h51m37.419318901s) [noise=true , address=77.90.31.15:4700 , piece_id=5745REMLDXFJCEN37YIQG3ERLKS7XSZ3FCYWH63NXDLNHEFBRSAQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: <nil> Dec 26 17:20:37.548255252 (505h51m37.419338287s) [seg_pos={0 0}, upload=443884]: Segment reached optimal threshold of 13 pieces. Dec 26 17:20:37.548427720 (505h51m37.419510765s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548434683 (505h51m37.419517719s) [noise=true , address=45.91.116.4:4940 , piece_id=H4AQ2VSADF5DE6RHVRPT3NX7YCP6K5UDWJNSDN44CR3QWYBA7SEQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548742390 (505h51m37.419825425s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 17:20:37.548749533 (505h51m37.419832578s) [noise=true , address=45.91.116.13:4156 , piece_id=QXA4CJZJEPS4YG46XQ4SWBHR3WN6H3XRB76YTN2K6LUNJB3AJDIQ, seg_pos={0 0}, upload=443884]: Long tail cancellation due to optimal threshold Dec 26 17:20:37.548766586 (505h51m37.419849621s) [seg_pos={0 0}, upload=443884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:20:37.548833213 (505h51m37.419916248s) [upload=443884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:20:37.548837932 (505h51m37.419920977s) [upload=443884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:20:37.848539526 (505h51m37.719622561s) [upload=443884]: Done uploading segments... Dec 26 17:20:37.848559173 (505h51m37.719642208s) [upload=443884]: Upload finished. err: <nil> Duration: 30.002943268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.......X....X........................................] duration: 42.949099327s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.949114237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.X] duration: 30.003005203s trace: https://jaeger.dev.storj.io/trace/27e58b05d1db4b7d FAIL (2025-12-26 17:20:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: LIBAKOX4NHGTRKVC2COCDQGJ6V3KAIRKJ4AD5LPACX3JYEECG7IA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: YGKPIGUVJFLWMBB3Q4247BG5YZUWWDJMSASAZHXEHTUH3YM3LRMA) context deadline exceeded ======LOGS====== pb.BatchRequestItem_ObjectBegin Dec 26 17:20:03.426830079 (505h51m3.297913124s) [upload=443879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:20:03.675409377 (505h51m3.546492422s) [seg_pos={0 0}, upload=443879]: Begin upload segment... Dec 26 17:20:03.675502916 (505h51m3.546585961s) [seg_pos={0 0}, upload=443879]: Done begin upload segment. Dec 26 17:20:03.675568159 (505h51m3.546651204s) [noise=true , address=145.79.227.11:4729 , piece_id=CHVNTT6GZ7HABYC5WP3TVNEPYYAQGXEYYKVOV37CEEAQZTQCY6OA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.678881238 (505h51m3.549964294s) [noise=true , address=145.79.227.51:4409 , piece_id=BFBZJ7GBZFLPLGJA6RPFSAKBPDFQGIEOWDDKGBJQJTYKJNUVQRQQ, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.679460852 (505h51m3.550543897s) [noise=true , address=45.91.116.6:4976 , piece_id=YGKPIGUVJFLWMBB3Q4247BG5YZUWWDJMSASAZHXEHTUH3YM3LRMA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.679948019 (505h51m3.551031064s) [noise=true , address=77.90.31.19:4844 , piece_id=H2IQZ42H3S423JESZ4DF4TKGKZCQKKFX6XQ5C3ASEVMWAZEXPWHA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.680288738 (505h51m3.551371783s) [noise=true , address=145.79.227.39:4015 , piece_id=VRUT6AOSFVHPDWAJTD2XJW4KCTHOPXQMOROJRTWKBT4TB2TSGTSQ, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.680349294 (505h51m3.551432339s) [noise=true , address=145.79.227.5:4222 , piece_id=7W47NLVZS5COOJ5GJNL36SGR7HSOEOO6LFDWINYPEG6HATUVRM7A, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.680659925 (505h51m3.55174297s) [noise=true , address=145.79.227.37:4565 , piece_id=SAOEH7FKG5R2IEF2T774U22EUAJKFNZWLBEXCOR2E4ZXPZYGZIYA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.681091547 (505h51m3.552174602s) [noise=true , address=45.91.116.4:4937 , piece_id=YAPQQF2E4M7HAQ2NF6ILI5YI376PHIBA3ORMU3O57DVBLXNECILA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.681140130 (505h51m3.552223165s) [noise=true , address=145.79.227.46:4602 , piece_id=JDUIVUJUI6LA2GC67EMIDQCRBMPJRAACVGKEJRT2KNVVCKTT4UMA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.681733099 (505h51m3.552816144s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEECLBZU7DGTIVPPOUOVSV6N4WSMLUHCEODBS442TAXTHWBJHXQ, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.682085300 (505h51m3.553168345s) [noise=true , address=145.79.227.7:4293 , piece_id=4SUHY6EXC25LGXYRW5CQL4N6KG4MXBCGN74RWKH5NQWPCR73LDCA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.682229744 (505h51m3.55331279s) [noise=true , address=77.90.31.18:4808 , piece_id=ENTVS5ZVLBHMUS7NJXL5MFE7YT3OXAIVUQ2NMXPGFD75LECP57BA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.682996434 (505h51m3.554079489s) [noise=true , address=45.91.116.5:4300 , piece_id=ZCWBQY6ZN5PPQLH7XQPBZOJ4TEAHZOIBV4TNUR3BKJY7SI7KA67Q, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.683972783 (505h51m3.555055828s) [noise=true , address=145.79.227.21:4475 , piece_id=UWUJJJS4EJY7YWURBD5R4O4D6AFYX4KBUSGIFT5BFIAORMJUCVDA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.684740285 (505h51m3.55582333s) [noise=true , address=45.91.116.12:4225 , piece_id=LIBAKOX4NHGTRKVC2COCDQGJ6V3KAIRKJ4AD5LPACX3JYEECG7IA, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:03.685489050 (505h51m3.556572095s) [noise=true , address=145.79.227.49:4337 , piece_id=ERL7FXSFSM5V2VPCFT2QQDG2YVJNLSIDAFDORVBKFEPEQ45PDHFQ, seg_pos={0 0}, upload=443879]: Uploading piece... Dec 26 17:20:05.342443290 (505h51m5.213526325s) [upload=443879]: Next returned nil segment Dec 26 17:20:05.342461315 (505h51m5.21354437s) [upload=443879]: Waiting for error group managing segments... Dec 26 17:20:05.729285656 (505h51m5.600368711s) [noise=true , address=145.79.227.7:4293 , piece_id=4SUHY6EXC25LGXYRW5CQL4N6KG4MXBCGN74RWKH5NQWPCR73LDCA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.729469356 (505h51m5.600552391s) [noise=true , address=145.79.227.46:4602 , piece_id=JDUIVUJUI6LA2GC67EMIDQCRBMPJRAACVGKEJRT2KNVVCKTT4UMA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.730893978 (505h51m5.601977013s) [noise=true , address=145.79.227.49:4337 , piece_id=ERL7FXSFSM5V2VPCFT2QQDG2YVJNLSIDAFDORVBKFEPEQ45PDHFQ, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.731987381 (505h51m5.603070426s) [noise=true , address=145.79.227.39:4015 , piece_id=VRUT6AOSFVHPDWAJTD2XJW4KCTHOPXQMOROJRTWKBT4TB2TSGTSQ, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.739018007 (505h51m5.610101053s) [noise=true , address=145.79.227.37:4565 , piece_id=SAOEH7FKG5R2IEF2T774U22EUAJKFNZWLBEXCOR2E4ZXPZYGZIYA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.739696851 (505h51m5.610779896s) [noise=true , address=145.79.227.11:4729 , piece_id=CHVNTT6GZ7HABYC5WP3TVNEPYYAQGXEYYKVOV37CEEAQZTQCY6OA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.740051967 (505h51m5.611135012s) [noise=true , address=145.79.227.21:4475 , piece_id=UWUJJJS4EJY7YWURBD5R4O4D6AFYX4KBUSGIFT5BFIAORMJUCVDA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.746252734 (505h51m5.617335779s) [noise=true , address=145.79.227.51:4409 , piece_id=BFBZJ7GBZFLPLGJA6RPFSAKBPDFQGIEOWDDKGBJQJTYKJNUVQRQQ, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.780526357 (505h51m5.651609402s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEECLBZU7DGTIVPPOUOVSV6N4WSMLUHCEODBS442TAXTHWBJHXQ, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.936823614 (505h51m5.807906659s) [noise=true , address=77.90.31.19:4844 , piece_id=H2IQZ42H3S423JESZ4DF4TKGKZCQKKFX6XQ5C3ASEVMWAZEXPWHA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.947641333 (505h51m5.818724398s) [noise=true , address=45.91.116.12:4225 , piece_id=LIBAKOX4NHGTRKVC2COCDQGJ6V3KAIRKJ4AD5LPACX3JYEECG7IA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:05.977245614 (505h51m5.84832865s) [noise=true , address=45.91.116.6:4976 , piece_id=YGKPIGUVJFLWMBB3Q4247BG5YZUWWDJMSASAZHXEHTUH3YM3LRMA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:06.185147035 (505h51m6.05623008s) [noise=true , address=145.79.227.5:4222 , piece_id=7W47NLVZS5COOJ5GJNL36SGR7HSOEOO6LFDWINYPEG6HATUVRM7A, seg_pos={0 0}, upload=443879]: Done uploading piece. err: <nil> Dec 26 17:20:06.185176141 (505h51m6.056259186s) [seg_pos={0 0}, upload=443879]: Segment reached optimal threshold of 13 pieces. Dec 26 17:20:06.185486621 (505h51m6.056569667s) [noise=true , address=45.91.116.4:4937 , piece_id=YAPQQF2E4M7HAQ2NF6ILI5YI376PHIBA3ORMU3O57DVBLXNECILA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40258->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40258->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40258->45.91.116.4:4937: use of closed network connection Dec 26 17:20:06.185496531 (505h51m6.056579566s) [noise=true , address=45.91.116.4:4937 , piece_id=YAPQQF2E4M7HAQ2NF6ILI5YI376PHIBA3ORMU3O57DVBLXNECILA, seg_pos={0 0}, upload=443879]: Long tail cancellation due to optimal threshold Dec 26 17:20:06.185865493 (505h51m6.056948548s) [noise=true , address=45.91.116.5:4300 , piece_id=ZCWBQY6ZN5PPQLH7XQPBZOJ4TEAHZOIBV4TNUR3BKJY7SI7KA67Q, seg_pos={0 0}, upload=443879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:36782->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:36782->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36782->45.91.116.5:4300: use of closed network connection Dec 26 17:20:06.185889699 (505h51m6.056972744s) [noise=true , address=45.91.116.5:4300 , piece_id=ZCWBQY6ZN5PPQLH7XQPBZOJ4TEAHZOIBV4TNUR3BKJY7SI7KA67Q, seg_pos={0 0}, upload=443879]: Long tail cancellation due to optimal threshold Dec 26 17:20:06.185936408 (505h51m6.057019453s) [noise=true , address=77.90.31.18:4808 , piece_id=ENTVS5ZVLBHMUS7NJXL5MFE7YT3OXAIVUQ2NMXPGFD75LECP57BA, seg_pos={0 0}, upload=443879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44020->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44020->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44020->77.90.31.18:4808: use of closed network connection Dec 26 17:20:06.185947980 (505h51m6.057031026s) [noise=true , address=77.90.31.18:4808 , piece_id=ENTVS5ZVLBHMUS7NJXL5MFE7YT3OXAIVUQ2NMXPGFD75LECP57BA, seg_pos={0 0}, upload=443879]: Long tail cancellation due to optimal threshold Dec 26 17:20:06.185973609 (505h51m6.057056654s) [seg_pos={0 0}, upload=443879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:20:06.186091594 (505h51m6.057174649s) [upload=443879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:20:06.186100771 (505h51m6.057183816s) [upload=443879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:20:06.490877109 (505h51m6.361960174s) [upload=443879]: Done uploading segments... Dec 26 17:20:06.490905793 (505h51m6.361988838s) [upload=443879]: Upload finished. err: <nil> Duration: 30.003032285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......X....X.......................................] duration: 46.649541721s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:20:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 46.649574264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.XXX...X........X...........................................] duration: 3.563661411s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:20:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563689603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X.] duration: 6.425738488s trace: https://jaeger.dev.storj.io/trace/4cc195fca15a7c47 FAIL (2025-12-26 17:19:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:18:37.777870471 (505h49m37.648953506s) [upload=443856]: Starting upload Dec 26 17:18:37.777923402 (505h49m37.649006437s) [upload=443856]: Done starting upload Dec 26 17:18:37.778097073 (505h49m37.649180108s) [upload=443856]: Uploading segments... Dec 26 17:18:37.778129054 (505h49m37.649212089s) [upload=443856]: Got next segment. Inline: false Dec 26 17:18:37.778265934 (505h49m37.649348969s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:18:37.778273369 (505h49m37.649356444s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:18:38.025749897 (505h49m37.896832932s) [seg_pos={0 0}, upload=443856]: Begin upload segment... Dec 26 17:18:38.025816554 (505h49m37.896899599s) [seg_pos={0 0}, upload=443856]: Done begin upload segment. Dec 26 17:18:38.025841882 (505h49m37.896924917s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025873222 (505h49m37.896956257s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025928557 (505h49m37.897011592s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026419021 (505h49m37.897502097s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026903673 (505h49m37.897986708s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027235215 (505h49m37.89831825s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027531730 (505h49m37.898614765s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027824537 (505h49m37.898907572s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028138094 (505h49m37.89922114s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028493782 (505h49m37.899576897s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029905250 (505h49m37.900988285s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029982947 (505h49m37.901065992s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.031149188 (505h49m37.902232233s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.032260233 (505h49m37.903343268s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.033446021 (505h49m37.904529066s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.034788017 (505h49m37.905871062s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:39.644387957 (505h49m39.515471002s) [upload=443856]: Next returned nil segment Dec 26 17:18:39.644405520 (505h49m39.515488555s) [upload=443856]: Waiting for error group managing segments... Dec 26 17:18:40.030562751 (505h49m39.901645786s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.031064616 (505h49m39.902147651s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.033601967 (505h49m39.904685002s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.034869291 (505h49m39.905952336s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.047423045 (505h49m39.91850609s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.053560312 (505h49m39.924643347s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.059360767 (505h49m39.930443812s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.066785935 (505h49m39.93786897s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.070927642 (505h49m39.942010717s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.270553644 (505h49m40.14163668s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.401567638 (505h49m40.272650673s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.403630647 (505h49m40.274713682s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623347386 (505h49m40.494430421s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623372052 (505h49m40.494455097s) [seg_pos={0 0}, upload=443856]: Segment reached optimal threshold of 13 pieces. Dec 26 17:18:40.623576141 (505h49m40.494659186s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623585970 (505h49m40.494669016s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623803705 (505h49m40.49488674s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623810177 (505h49m40.494893212s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623827299 (505h49m40.494910334s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623846997 (505h49m40.494930042s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623871514 (505h49m40.494954559s) [seg_pos={0 0}, upload=443856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:18:40.623992634 (505h49m40.495075669s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:18:40.623999096 (505h49m40.495082141s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:18:40.914858874 (505h49m40.785941909s) [upload=443856]: Done uploading segments... Dec 26 17:18:40.914878332 (505h49m40.785961367s) [upload=443856]: Upload finished. err: <nil> Duration: 6.425763315s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..XXX...X........X..........................................] duration: 3.589037824s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:19:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5890623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......X....X......................................] duration: 26.602469261s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:19:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 26.602490131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.X] duration: 30.003418542s trace: https://jaeger.dev.storj.io/trace/4cc195fca15a7c47 FAIL (2025-12-26 17:19:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:18:37.777870471 (505h49m37.648953506s) [upload=443856]: Starting upload Dec 26 17:18:37.777923402 (505h49m37.649006437s) [upload=443856]: Done starting upload Dec 26 17:18:37.778097073 (505h49m37.649180108s) [upload=443856]: Uploading segments... Dec 26 17:18:37.778129054 (505h49m37.649212089s) [upload=443856]: Got next segment. Inline: false Dec 26 17:18:37.778265934 (505h49m37.649348969s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:18:37.778273369 (505h49m37.649356444s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:18:38.025749897 (505h49m37.896832932s) [seg_pos={0 0}, upload=443856]: Begin upload segment... Dec 26 17:18:38.025816554 (505h49m37.896899599s) [seg_pos={0 0}, upload=443856]: Done begin upload segment. Dec 26 17:18:38.025841882 (505h49m37.896924917s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025873222 (505h49m37.896956257s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.025928557 (505h49m37.897011592s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026419021 (505h49m37.897502097s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.026903673 (505h49m37.897986708s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027235215 (505h49m37.89831825s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027531730 (505h49m37.898614765s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.027824537 (505h49m37.898907572s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028138094 (505h49m37.89922114s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.028493782 (505h49m37.899576897s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029905250 (505h49m37.900988285s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.029982947 (505h49m37.901065992s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.031149188 (505h49m37.902232233s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.032260233 (505h49m37.903343268s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.033446021 (505h49m37.904529066s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:38.034788017 (505h49m37.905871062s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Uploading piece... Dec 26 17:18:39.644387957 (505h49m39.515471002s) [upload=443856]: Next returned nil segment Dec 26 17:18:39.644405520 (505h49m39.515488555s) [upload=443856]: Waiting for error group managing segments... Dec 26 17:18:40.030562751 (505h49m39.901645786s) [noise=true , address=145.79.227.19:4077 , piece_id=S5KVS6MLQYT22D35CBUCM7ODNS3DKFYUACAD6LXCXT54AOZKVIZQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.031064616 (505h49m39.902147651s) [noise=true , address=145.79.227.45:4231 , piece_id=CUL63NM45AT7HOY56GVU43MDC42DJSC2YVKCX55BW7VXYZQ3VTDA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.033601967 (505h49m39.904685002s) [noise=true , address=145.79.227.41:4087 , piece_id=JZLWW6Q5QQT2VNDV2H2QSFGV6L6XWUE7BJLSFGYIDOEI3YYN6DNQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.034869291 (505h49m39.905952336s) [noise=true , address=145.79.227.36:4519 , piece_id=NHS45M63RWZFBJJDJZ6ZWHKYQKRMTPHHP4KO6OQHV2KTO6CUPK2A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.047423045 (505h49m39.91850609s) [noise=true , address=145.79.227.50:4374 , piece_id=2J4Y6UHACJMVM5N5UHPL6DZDEH7JSNIX5YITSDX2G373B63643VQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.053560312 (505h49m39.924643347s) [noise=true , address=145.79.227.11:4729 , piece_id=WQLMCI436FX7EWCRAKOVOLCFSPPZ2CF44UT6BJPKVBVJYLWH62GQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.059360767 (505h49m39.930443812s) [noise=true , address=145.79.227.42:4123 , piece_id=E25UYPCGLURGHMDHITO5V77I2SYK55TW2ULX66MPPKHKTARUZAVQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.066785935 (505h49m39.93786897s) [noise=true , address=145.79.227.49:4338 , piece_id=HZNN23DKIIILRPSROKIMVUEF4WFUHCIOWRJOFVMNYFL77E2FGPEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.070927642 (505h49m39.942010717s) [noise=true , address=145.79.227.14:4839 , piece_id=Z5RMRSNLJ3AGM6JRI6HU2GOMPX3D6U6WXF4TW3BKWRCULQSPQPJA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.270553644 (505h49m40.14163668s) [noise=true , address=45.91.116.4:4939 , piece_id=7KT33KC3VR32A5EQSW6ZJ2H65NO67EV2RTWCXWSDEBLLOL2YCDCA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.401567638 (505h49m40.272650673s) [noise=true , address=45.91.116.6:4975 , piece_id=RLKGVZ4HMHRNQS544GQQ7EVX5PM2WSMVUF547WF57UJ7UOJP2TGA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.403630647 (505h49m40.274713682s) [noise=true , address=45.91.116.14:4192 , piece_id=ATUU5ST2DN3M6AJAW4HDH7AK7KTJ7AD3IPPIOJG2K7T4URMJDLFQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623347386 (505h49m40.494430421s) [noise=true , address=45.91.116.10:4082 , piece_id=PESG4F7BP2EZHTEGDEQ4OBACCLPTK7TXVEKF745KFMY2WU27DCAA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: <nil> Dec 26 17:18:40.623372052 (505h49m40.494455097s) [seg_pos={0 0}, upload=443856]: Segment reached optimal threshold of 13 pieces. Dec 26 17:18:40.623576141 (505h49m40.494659186s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623585970 (505h49m40.494669016s) [noise=true , address=145.79.227.44:4195 , piece_id=ADAR2G2KRCCIQFEIDKGUYEHG72NG2EG55D7SUGBPHX2SDANMMVEQ, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623803705 (505h49m40.49488674s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623810177 (505h49m40.494893212s) [noise=true , address=145.79.227.43:4159 , piece_id=DKVSSDGLOWE6NP6XYWBEMHHTBX2AEXWLYA7OYOHTMSOWFSVY4M4A, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623827299 (505h49m40.494910334s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 17:18:40.623846997 (505h49m40.494930042s) [noise=true , address=45.91.116.9:4262 , piece_id=AMYSLWZNQUWQJ5DV4X7PULOEAR55DYO2X4LADKWFMSWBIV6K5WXA, seg_pos={0 0}, upload=443856]: Long tail cancellation due to optimal threshold Dec 26 17:18:40.623871514 (505h49m40.494954559s) [seg_pos={0 0}, upload=443856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:18:40.623992634 (505h49m40.495075669s) [upload=443856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:18:40.623999096 (505h49m40.495082141s) [upload=443856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:18:40.914858874 (505h49m40.785941909s) [upload=443856]: Done uploading segments... Dec 26 17:18:40.914878332 (505h49m40.785961367s) [upload=443856]: Upload finished. err: <nil> Duration: 30.003442388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......X....X.....................................] duration: 54.719769566s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 54.719799121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...XXX...X........X.........................................] duration: 3.507813589s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:18:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507837985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....XXX...X........X........................................] duration: 3.604349189s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:17:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604375369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X.] duration: 6.187458491s trace: https://jaeger.dev.storj.io/trace/747586e3aa30dcfa FAIL (2025-12-26 17:17:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA) context deadline exceeded ======LOGS====== Dec 26 17:16:43.166520031 (505h47m43.037603066s) [upload=443828]: Starting upload Dec 26 17:16:43.166580586 (505h47m43.037663622s) [upload=443828]: Done starting upload Dec 26 17:16:43.166652694 (505h47m43.037735739s) [upload=443828]: Uploading segments... Dec 26 17:16:43.166810954 (505h47m43.03789399s) [upload=443828]: Got next segment. Inline: false Dec 26 17:16:43.166849488 (505h47m43.037932533s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:16:43.166854188 (505h47m43.037937233s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:16:43.426770333 (505h47m43.297853398s) [seg_pos={0 0}, upload=443828]: Begin upload segment... Dec 26 17:16:43.426845216 (505h47m43.297928261s) [seg_pos={0 0}, upload=443828]: Done begin upload segment. Dec 26 17:16:43.426880583 (505h47m43.297963628s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433145322 (505h47m43.304228357s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433884860 (505h47m43.304967895s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.435975611 (505h47m43.307058646s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.438307441 (505h47m43.309390487s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.440534272 (505h47m43.311617317s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.442626315 (505h47m43.31370937s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.445255162 (505h47m43.316338197s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.446248623 (505h47m43.317331678s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.447418741 (505h47m43.318501786s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448881366 (505h47m43.319964401s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448983050 (505h47m43.320066125s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.450465342 (505h47m43.321548387s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.451848827 (505h47m43.322931872s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.452402511 (505h47m43.323485576s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.454342485 (505h47m43.32542552s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:45.157230350 (505h47m45.028313385s) [upload=443828]: Next returned nil segment Dec 26 17:16:45.157243164 (505h47m45.028326199s) [upload=443828]: Waiting for error group managing segments... Dec 26 17:16:45.430886079 (505h47m45.301969124s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.445045310 (505h47m45.316128345s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.451806074 (505h47m45.322889119s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.452867274 (505h47m45.323950319s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.456499641 (505h47m45.327582686s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.462561865 (505h47m45.3336449s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.470207082 (505h47m45.341290137s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.473631944 (505h47m45.344714989s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.474290298 (505h47m45.345373333s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.496098745 (505h47m45.36718178s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.758545770 (505h47m45.629628815s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.799247743 (505h47m45.670330778s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.979342215 (505h47m45.85042526s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 17:16:46.329608904 (505h47m46.200691949s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:46.329631226 (505h47m46.200714271s) [seg_pos={0 0}, upload=443828]: Segment reached optimal threshold of 13 pieces. Dec 26 17:16:46.329863578 (505h47m46.200946614s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 17:16:46.329873096 (505h47m46.200956141s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329908023 (505h47m46.200991058s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection Dec 26 17:16:46.329916971 (505h47m46.201000006s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329946526 (505h47m46.201029571s) [seg_pos={0 0}, upload=443828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:16:46.330107573 (505h47m46.201190608s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:16:46.330116700 (505h47m46.201199735s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:16:46.631634422 (505h47m46.502717467s) [upload=443828]: Done uploading segments... Dec 26 17:16:46.631658008 (505h47m46.502741043s) [upload=443828]: Upload finished. err: <nil> Duration: 6.187481283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......X....X....................................] duration: 49.487538093s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 49.487555887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....XXX...X........X.......................................] duration: 3.497761838s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:17:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497802135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....X] duration: 30.002123195s trace: https://jaeger.dev.storj.io/trace/747586e3aa30dcfa FAIL (2025-12-26 17:17:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA) context deadline exceeded ======LOGS====== Dec 26 17:16:43.166520031 (505h47m43.037603066s) [upload=443828]: Starting upload Dec 26 17:16:43.166580586 (505h47m43.037663622s) [upload=443828]: Done starting upload Dec 26 17:16:43.166652694 (505h47m43.037735739s) [upload=443828]: Uploading segments... Dec 26 17:16:43.166810954 (505h47m43.03789399s) [upload=443828]: Got next segment. Inline: false Dec 26 17:16:43.166849488 (505h47m43.037932533s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:16:43.166854188 (505h47m43.037937233s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:16:43.426770333 (505h47m43.297853398s) [seg_pos={0 0}, upload=443828]: Begin upload segment... Dec 26 17:16:43.426845216 (505h47m43.297928261s) [seg_pos={0 0}, upload=443828]: Done begin upload segment. Dec 26 17:16:43.426880583 (505h47m43.297963628s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433145322 (505h47m43.304228357s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.433884860 (505h47m43.304967895s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.435975611 (505h47m43.307058646s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.438307441 (505h47m43.309390487s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.440534272 (505h47m43.311617317s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.442626315 (505h47m43.31370937s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.445255162 (505h47m43.316338197s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.446248623 (505h47m43.317331678s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.447418741 (505h47m43.318501786s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448881366 (505h47m43.319964401s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.448983050 (505h47m43.320066125s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.450465342 (505h47m43.321548387s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.451848827 (505h47m43.322931872s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.452402511 (505h47m43.323485576s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:43.454342485 (505h47m43.32542552s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Uploading piece... Dec 26 17:16:45.157230350 (505h47m45.028313385s) [upload=443828]: Next returned nil segment Dec 26 17:16:45.157243164 (505h47m45.028326199s) [upload=443828]: Waiting for error group managing segments... Dec 26 17:16:45.430886079 (505h47m45.301969124s) [noise=true , address=145.79.227.37:4565 , piece_id=OJYFTYNQ7R32GYXHBSEJUF4DESGQA2XBAO4YC2MFSEDLRXLF7YDQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.445045310 (505h47m45.316128345s) [noise=true , address=145.79.227.14:4839 , piece_id=BQN5YJF75FCDYZSOABB7KQMMDRVYGJYTYZLVY6XO4PQV3PEIOPCA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.451806074 (505h47m45.322889119s) [noise=true , address=145.79.227.38:4977 , piece_id=547C2FD67QFDJA33GBNVNFONRKODVCG7IEBQLGZEEWDLGFLW5RVQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.452867274 (505h47m45.323950319s) [noise=true , address=145.79.227.51:4410 , piece_id=QFWMUR3PDRDXBOZ7IBEGHGV6JRGP4BUPJARWTZWGF5HSIIOXYRSA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.456499641 (505h47m45.327582686s) [noise=true , address=145.79.227.11:4732 , piece_id=NDCWMGP2DESMSYE6UG6XPFEBBC5Z4A5TUL3M26HKXYXZG2BKS2EQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.462561865 (505h47m45.3336449s) [noise=true , address=145.79.227.42:4123 , piece_id=HACOMIPTRXR5YMK6F2HUY2WFGQCI5WN3PZOPV5ZFJ7FCVMZKWLJQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.470207082 (505h47m45.341290137s) [noise=true , address=145.79.227.45:4231 , piece_id=EZJ42YYXFQI6FLE3PGKVSN7KEEKOFDLMB7GNG56UFWZ2AOC2DTGQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.473631944 (505h47m45.344714989s) [noise=true , address=145.79.227.52:4446 , piece_id=EHPJHELMJJZ4Q4VOXJK7SLR4ULP3NJD7SV52MMA3AY2TFANCMVOQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.474290298 (505h47m45.345373333s) [noise=true , address=145.79.227.41:4087 , piece_id=OAT64JX35XOPADOYBFGUGH6SA4NJI3RX3TANPJK7DAFSRDEC6DRQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.496098745 (505h47m45.36718178s) [noise=true , address=145.79.227.47:4267 , piece_id=5MTAVUIJAM2NGZOR25Q3HRFEM3VVJZTGMPL6TQI3SKQUYQ6RA3AA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.758545770 (505h47m45.629628815s) [noise=true , address=45.91.116.4:4937 , piece_id=WAMMVLVTRNETSOVAOUOL6N4OFIPOXT3VSYVHBANA2OT6YMIBFHLA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.799247743 (505h47m45.670330778s) [noise=true , address=45.91.116.14:4190 , piece_id=FX2UMU5GNHTADF6ZB6PZXOGYFBRR35FINUBRFHHLH565QFBH4XTA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:45.979342215 (505h47m45.85042526s) [noise=true , address=77.90.31.11:4554 , piece_id=CLM5TWTTDOBJBTUZH7SOP7CYILE5N6Z45AELUGYZEECYD3T4NBMA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 17:16:46.329608904 (505h47m46.200691949s) [noise=true , address=45.91.116.6:4974 , piece_id=4CWQBIGTBNQDC3WX3FGRVZWMKCYM2EJPGX4NV5EE3HWF6LR5KZXQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: <nil> Dec 26 17:16:46.329631226 (505h47m46.200714271s) [seg_pos={0 0}, upload=443828]: Segment reached optimal threshold of 13 pieces. Dec 26 17:16:46.329863578 (505h47m46.200946614s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 17:16:46.329873096 (505h47m46.200956141s) [noise=true , address=145.79.227.40:4051 , piece_id=E5WZZPYNUVJEHQBYCKLLKHA7OQ6O6G3PCNDJCWWBVRO7K3GJSCKA, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329908023 (505h47m46.200991058s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38846->145.79.227.7:4293: use of closed network connection Dec 26 17:16:46.329916971 (505h47m46.201000006s) [noise=true , address=145.79.227.7:4293 , piece_id=3TJSYKJWV4CSQJM25P7LVI4VFMCWO2MXK2VKQ35VMCUOOBF56GQQ, seg_pos={0 0}, upload=443828]: Long tail cancellation due to optimal threshold Dec 26 17:16:46.329946526 (505h47m46.201029571s) [seg_pos={0 0}, upload=443828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:16:46.330107573 (505h47m46.201190608s) [upload=443828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:16:46.330116700 (505h47m46.201199735s) [upload=443828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:16:46.631634422 (505h47m46.502717467s) [upload=443828]: Done uploading segments... Dec 26 17:16:46.631658008 (505h47m46.502741043s) [upload=443828]: Upload finished. err: <nil> Duration: 30.002305051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X....X...................................] duration: 27.789723662s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:16:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 27.789743168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......XXX...X........X......................................] duration: 4.392848019s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:16:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.392870173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X....] duration: 16.035107872s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 16.035135233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X....X..................................] duration: 39.408699659s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 39.408729024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X......XXX...X........X.....................................] duration: 3.445370075s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:15:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445399351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X...] duration: 6.230150855s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:15:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 6.230183606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......X....X.................................] duration: 45.043686444s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 45.043717793s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)