Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [....XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.] duration: 6.346763847s trace: https://jaeger.dev.storj.io/trace/7d71ed27650d4659 FAIL (2025-11-07 18:06:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: QNUH2VD34HAQ74NYWQHQNLVLUDOCLKQAM66JSLBBR4S5NBJ7XPJA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: D3YTCMXOKTWOQRSRAEQ3UAUNSGUMSYWZDQVKIHDADRKMUDKKD4MQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: AITHF2ZRZVL2Q2IHQSEPNIE2C2MWVM2YXNVITSAKMTYX657FV7FA) context deadline exceeded ======LOGS====== 8655016 (198h25m28.404503769s) [upload=174871]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:05:56.957321803 (198h25m28.653170566s) [seg_pos={0 0}, upload=174871]: Begin upload segment... Nov 7 18:05:56.957442823 (198h25m28.653291576s) [seg_pos={0 0}, upload=174871]: Done begin upload segment. Nov 7 18:05:56.957453494 (198h25m28.653302257s) [noise=true , address=145.79.227.17:4972 , piece_id=7Z7HRZCDUJFDNDA3UOQD6Z4QQ7RY7PHDAWM76BCUF5UINF74ECFA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.957488240 (198h25m28.653336993s) [noise=true , address=77.90.31.19:4843 , piece_id=DUIPLQBOUDAOZ2HUDET7ZJFXVC3V46JXRZD442JUKZS5VCGFTDGQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.958211136 (198h25m28.654059889s) [noise=true , address=45.91.116.13:4153 , piece_id=QNUH2VD34HAQ74NYWQHQNLVLUDOCLKQAM66JSLBBR4S5NBJ7XPJA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.958622349 (198h25m28.654471102s) [noise=true , address=77.90.31.11:4555 , piece_id=DMFSAYIRFEUC4YZFECB3RCMHCY4AY6H3PJOSWEY4X5N5FYRKWZRQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.959162839 (198h25m28.655011592s) [noise=true , address=145.79.227.22:4510 , piece_id=E2J3IJA273MMM75UYJLSFHWVHBDI2EDAOYM3CHCF6FZGKUNCDN6Q, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.959617344 (198h25m28.655466097s) [noise=true , address=145.79.227.25:4618 , piece_id=LSQTGH4XAZ3PYNSMJ6UWSZBPKAFX2KA7EI4OR6QHCD5Q65IUCVPQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.960223318 (198h25m28.656072071s) [noise=true , address=45.91.116.7:4010 , piece_id=D3YTCMXOKTWOQRSRAEQ3UAUNSGUMSYWZDQVKIHDADRKMUDKKD4MQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.960742957 (198h25m28.65659171s) [noise=true , address=45.91.116.6:4973 , piece_id=BPJCLQ2MNJHJGSATR3PVHHDCQN7A3BA4WZLCJQYZJFB4APFXGDSQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.961361736 (198h25m28.657210499s) [noise=true , address=145.79.227.23:4546 , piece_id=AOW7DKT7BRK6FDQ6B6P5I2SVDO4OOQRYG6BY4EJZKLREP6JXCMTQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.962201424 (198h25m28.658050178s) [noise=true , address=145.79.227.28:4727 , piece_id=KGXUSZSRZWPUH6GFNZG45WA7FAQCNKAANJOQK3NRMA637QFXQT2A, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.964174893 (198h25m28.660023646s) [noise=true , address=45.91.116.14:4190 , piece_id=KFLJSQC26OTMY4MGM4GDC4TPODPPM2RQXCB2OPKFLXM7WF572FJQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.966018062 (198h25m28.661866815s) [noise=true , address=145.79.227.20:4437 , piece_id=4TK7RPCKTTQWIZ7W44C44IAOZIIAU2BSCIZ42TWZZBFNFOQ35GWQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.967791780 (198h25m28.663640543s) [noise=true , address=145.79.227.26:4654 , piece_id=HWT5GP4LFV5YSWTZWHE42SCKH7VHG6OL37HICFVITIN3E2FSNZ7A, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.969547663 (198h25m28.665396426s) [noise=true , address=45.91.116.10:4083 , piece_id=AITHF2ZRZVL2Q2IHQSEPNIE2C2MWVM2YXNVITSAKMTYX657FV7FA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.972237085 (198h25m28.668085838s) [noise=true , address=145.79.227.9:4367 , piece_id=2N7B6BFQYKP47FETWPTGLFMXJFDLTYT4EE7XSEL6MMTP47SHG4DA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.975279038 (198h25m28.671127801s) [noise=true , address=77.90.31.16:4735 , piece_id=VEW2DC562VHY2L63V7RS6Y7GB343AX7WJNF4UK5A6ZIZOUBDYKJQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:58.636197873 (198h25m30.332046626s) [upload=174871]: Next returned nil segment Nov 7 18:05:58.636215928 (198h25m30.332064681s) [upload=174871]: Waiting for error group managing segments... Nov 7 18:05:58.914212636 (198h25m30.610061379s) [noise=true , address=145.79.227.20:4437 , piece_id=4TK7RPCKTTQWIZ7W44C44IAOZIIAU2BSCIZ42TWZZBFNFOQ35GWQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.914693201 (198h25m30.610541954s) [noise=true , address=145.79.227.25:4618 , piece_id=LSQTGH4XAZ3PYNSMJ6UWSZBPKAFX2KA7EI4OR6QHCD5Q65IUCVPQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.969273284 (198h25m30.665122037s) [noise=true , address=145.79.227.26:4654 , piece_id=HWT5GP4LFV5YSWTZWHE42SCKH7VHG6OL37HICFVITIN3E2FSNZ7A, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.983855140 (198h25m30.679703893s) [noise=true , address=145.79.227.22:4510 , piece_id=E2J3IJA273MMM75UYJLSFHWVHBDI2EDAOYM3CHCF6FZGKUNCDN6Q, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.995772904 (198h25m30.691621657s) [noise=true , address=145.79.227.28:4727 , piece_id=KGXUSZSRZWPUH6GFNZG45WA7FAQCNKAANJOQK3NRMA637QFXQT2A, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.999505191 (198h25m30.695353944s) [noise=true , address=145.79.227.9:4367 , piece_id=2N7B6BFQYKP47FETWPTGLFMXJFDLTYT4EE7XSEL6MMTP47SHG4DA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.024561002 (198h25m30.720409795s) [noise=true , address=145.79.227.17:4972 , piece_id=7Z7HRZCDUJFDNDA3UOQD6Z4QQ7RY7PHDAWM76BCUF5UINF74ECFA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.183808154 (198h25m30.879656908s) [noise=true , address=45.91.116.6:4973 , piece_id=BPJCLQ2MNJHJGSATR3PVHHDCQN7A3BA4WZLCJQYZJFB4APFXGDSQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.204036905 (198h25m30.899885648s) [noise=true , address=45.91.116.7:4010 , piece_id=D3YTCMXOKTWOQRSRAEQ3UAUNSGUMSYWZDQVKIHDADRKMUDKKD4MQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.213347674 (198h25m30.909196417s) [noise=true , address=45.91.116.13:4153 , piece_id=QNUH2VD34HAQ74NYWQHQNLVLUDOCLKQAM66JSLBBR4S5NBJ7XPJA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.235982496 (198h25m30.931831249s) [noise=true , address=77.90.31.19:4843 , piece_id=DUIPLQBOUDAOZ2HUDET7ZJFXVC3V46JXRZD442JUKZS5VCGFTDGQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.254285809 (198h25m30.950134563s) [noise=true , address=145.79.227.23:4546 , piece_id=AOW7DKT7BRK6FDQ6B6P5I2SVDO4OOQRYG6BY4EJZKLREP6JXCMTQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.254463479 (198h25m30.950312232s) [noise=true , address=45.91.116.10:4083 , piece_id=AITHF2ZRZVL2Q2IHQSEPNIE2C2MWVM2YXNVITSAKMTYX657FV7FA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.254472265 (198h25m30.950321018s) [seg_pos={0 0}, upload=174871]: Segment reached optimal threshold of 13 pieces. Nov 7 18:05:59.254701491 (198h25m30.950550234s) [noise=true , address=45.91.116.14:4190 , piece_id=KFLJSQC26OTMY4MGM4GDC4TPODPPM2RQXCB2OPKFLXM7WF572FJQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:32854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:32854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32854->45.91.116.14:4190: use of closed network connection Nov 7 18:05:59.254711872 (198h25m30.950560665s) [noise=true , address=45.91.116.14:4190 , piece_id=KFLJSQC26OTMY4MGM4GDC4TPODPPM2RQXCB2OPKFLXM7WF572FJQ, seg_pos={0 0}, upload=174871]: Long tail cancellation due to optimal threshold Nov 7 18:05:59.254734474 (198h25m30.950583237s) [noise=true , address=77.90.31.11:4555 , piece_id=DMFSAYIRFEUC4YZFECB3RCMHCY4AY6H3PJOSWEY4X5N5FYRKWZRQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:36358->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:36358->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36358->77.90.31.11:4555: use of closed network connection Nov 7 18:05:59.254745345 (198h25m30.950594088s) [noise=true , address=77.90.31.11:4555 , piece_id=DMFSAYIRFEUC4YZFECB3RCMHCY4AY6H3PJOSWEY4X5N5FYRKWZRQ, seg_pos={0 0}, upload=174871]: Long tail cancellation due to optimal threshold Nov 7 18:05:59.254860735 (198h25m30.950709478s) [noise=true , address=77.90.31.16:4735 , piece_id=VEW2DC562VHY2L63V7RS6Y7GB343AX7WJNF4UK5A6ZIZOUBDYKJQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33180->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33180->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33180->77.90.31.16:4735: use of closed network connection Nov 7 18:05:59.254870764 (198h25m30.950719517s) [noise=true , address=77.90.31.16:4735 , piece_id=VEW2DC562VHY2L63V7RS6Y7GB343AX7WJNF4UK5A6ZIZOUBDYKJQ, seg_pos={0 0}, upload=174871]: Long tail cancellation due to optimal threshold Nov 7 18:05:59.255114588 (198h25m30.950963341s) [seg_pos={0 0}, upload=174871]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:05:59.255230920 (198h25m30.951079673s) [upload=174871]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:05:59.255238073 (198h25m30.951086826s) [upload=174871]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:05:59.567673149 (198h25m31.263521903s) [upload=174871]: Done uploading segments... Nov 7 18:05:59.567694030 (198h25m31.263542773s) [upload=174871]: Upload finished. err: <nil> Duration:* 6.346795638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.568633262s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:06:29 UTC) orbiter warning: context deadline exceeded Duration: 3.568659641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.....XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX] duration: 30.004469006s trace: https://jaeger.dev.storj.io/trace/7d71ed27650d4659 FAIL (2025-11-07 18:06:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: QNUH2VD34HAQ74NYWQHQNLVLUDOCLKQAM66JSLBBR4S5NBJ7XPJA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: D3YTCMXOKTWOQRSRAEQ3UAUNSGUMSYWZDQVKIHDADRKMUDKKD4MQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: AITHF2ZRZVL2Q2IHQSEPNIE2C2MWVM2YXNVITSAKMTYX657FV7FA) context deadline exceeded ======LOGS====== 8655016 (198h25m28.404503769s) [upload=174871]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:05:56.957321803 (198h25m28.653170566s) [seg_pos={0 0}, upload=174871]: Begin upload segment... Nov 7 18:05:56.957442823 (198h25m28.653291576s) [seg_pos={0 0}, upload=174871]: Done begin upload segment. Nov 7 18:05:56.957453494 (198h25m28.653302257s) [noise=true , address=145.79.227.17:4972 , piece_id=7Z7HRZCDUJFDNDA3UOQD6Z4QQ7RY7PHDAWM76BCUF5UINF74ECFA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.957488240 (198h25m28.653336993s) [noise=true , address=77.90.31.19:4843 , piece_id=DUIPLQBOUDAOZ2HUDET7ZJFXVC3V46JXRZD442JUKZS5VCGFTDGQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.958211136 (198h25m28.654059889s) [noise=true , address=45.91.116.13:4153 , piece_id=QNUH2VD34HAQ74NYWQHQNLVLUDOCLKQAM66JSLBBR4S5NBJ7XPJA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.958622349 (198h25m28.654471102s) [noise=true , address=77.90.31.11:4555 , piece_id=DMFSAYIRFEUC4YZFECB3RCMHCY4AY6H3PJOSWEY4X5N5FYRKWZRQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.959162839 (198h25m28.655011592s) [noise=true , address=145.79.227.22:4510 , piece_id=E2J3IJA273MMM75UYJLSFHWVHBDI2EDAOYM3CHCF6FZGKUNCDN6Q, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.959617344 (198h25m28.655466097s) [noise=true , address=145.79.227.25:4618 , piece_id=LSQTGH4XAZ3PYNSMJ6UWSZBPKAFX2KA7EI4OR6QHCD5Q65IUCVPQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.960223318 (198h25m28.656072071s) [noise=true , address=45.91.116.7:4010 , piece_id=D3YTCMXOKTWOQRSRAEQ3UAUNSGUMSYWZDQVKIHDADRKMUDKKD4MQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.960742957 (198h25m28.65659171s) [noise=true , address=45.91.116.6:4973 , piece_id=BPJCLQ2MNJHJGSATR3PVHHDCQN7A3BA4WZLCJQYZJFB4APFXGDSQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.961361736 (198h25m28.657210499s) [noise=true , address=145.79.227.23:4546 , piece_id=AOW7DKT7BRK6FDQ6B6P5I2SVDO4OOQRYG6BY4EJZKLREP6JXCMTQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.962201424 (198h25m28.658050178s) [noise=true , address=145.79.227.28:4727 , piece_id=KGXUSZSRZWPUH6GFNZG45WA7FAQCNKAANJOQK3NRMA637QFXQT2A, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.964174893 (198h25m28.660023646s) [noise=true , address=45.91.116.14:4190 , piece_id=KFLJSQC26OTMY4MGM4GDC4TPODPPM2RQXCB2OPKFLXM7WF572FJQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.966018062 (198h25m28.661866815s) [noise=true , address=145.79.227.20:4437 , piece_id=4TK7RPCKTTQWIZ7W44C44IAOZIIAU2BSCIZ42TWZZBFNFOQ35GWQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.967791780 (198h25m28.663640543s) [noise=true , address=145.79.227.26:4654 , piece_id=HWT5GP4LFV5YSWTZWHE42SCKH7VHG6OL37HICFVITIN3E2FSNZ7A, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.969547663 (198h25m28.665396426s) [noise=true , address=45.91.116.10:4083 , piece_id=AITHF2ZRZVL2Q2IHQSEPNIE2C2MWVM2YXNVITSAKMTYX657FV7FA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.972237085 (198h25m28.668085838s) [noise=true , address=145.79.227.9:4367 , piece_id=2N7B6BFQYKP47FETWPTGLFMXJFDLTYT4EE7XSEL6MMTP47SHG4DA, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:56.975279038 (198h25m28.671127801s) [noise=true , address=77.90.31.16:4735 , piece_id=VEW2DC562VHY2L63V7RS6Y7GB343AX7WJNF4UK5A6ZIZOUBDYKJQ, seg_pos={0 0}, upload=174871]: Uploading piece... Nov 7 18:05:58.636197873 (198h25m30.332046626s) [upload=174871]: Next returned nil segment Nov 7 18:05:58.636215928 (198h25m30.332064681s) [upload=174871]: Waiting for error group managing segments... Nov 7 18:05:58.914212636 (198h25m30.610061379s) [noise=true , address=145.79.227.20:4437 , piece_id=4TK7RPCKTTQWIZ7W44C44IAOZIIAU2BSCIZ42TWZZBFNFOQ35GWQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.914693201 (198h25m30.610541954s) [noise=true , address=145.79.227.25:4618 , piece_id=LSQTGH4XAZ3PYNSMJ6UWSZBPKAFX2KA7EI4OR6QHCD5Q65IUCVPQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.969273284 (198h25m30.665122037s) [noise=true , address=145.79.227.26:4654 , piece_id=HWT5GP4LFV5YSWTZWHE42SCKH7VHG6OL37HICFVITIN3E2FSNZ7A, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.983855140 (198h25m30.679703893s) [noise=true , address=145.79.227.22:4510 , piece_id=E2J3IJA273MMM75UYJLSFHWVHBDI2EDAOYM3CHCF6FZGKUNCDN6Q, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.995772904 (198h25m30.691621657s) [noise=true , address=145.79.227.28:4727 , piece_id=KGXUSZSRZWPUH6GFNZG45WA7FAQCNKAANJOQK3NRMA637QFXQT2A, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:58.999505191 (198h25m30.695353944s) [noise=true , address=145.79.227.9:4367 , piece_id=2N7B6BFQYKP47FETWPTGLFMXJFDLTYT4EE7XSEL6MMTP47SHG4DA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.024561002 (198h25m30.720409795s) [noise=true , address=145.79.227.17:4972 , piece_id=7Z7HRZCDUJFDNDA3UOQD6Z4QQ7RY7PHDAWM76BCUF5UINF74ECFA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.183808154 (198h25m30.879656908s) [noise=true , address=45.91.116.6:4973 , piece_id=BPJCLQ2MNJHJGSATR3PVHHDCQN7A3BA4WZLCJQYZJFB4APFXGDSQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.204036905 (198h25m30.899885648s) [noise=true , address=45.91.116.7:4010 , piece_id=D3YTCMXOKTWOQRSRAEQ3UAUNSGUMSYWZDQVKIHDADRKMUDKKD4MQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.213347674 (198h25m30.909196417s) [noise=true , address=45.91.116.13:4153 , piece_id=QNUH2VD34HAQ74NYWQHQNLVLUDOCLKQAM66JSLBBR4S5NBJ7XPJA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.235982496 (198h25m30.931831249s) [noise=true , address=77.90.31.19:4843 , piece_id=DUIPLQBOUDAOZ2HUDET7ZJFXVC3V46JXRZD442JUKZS5VCGFTDGQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.254285809 (198h25m30.950134563s) [noise=true , address=145.79.227.23:4546 , piece_id=AOW7DKT7BRK6FDQ6B6P5I2SVDO4OOQRYG6BY4EJZKLREP6JXCMTQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.254463479 (198h25m30.950312232s) [noise=true , address=45.91.116.10:4083 , piece_id=AITHF2ZRZVL2Q2IHQSEPNIE2C2MWVM2YXNVITSAKMTYX657FV7FA, seg_pos={0 0}, upload=174871]: Done uploading piece. err: <nil> Nov 7 18:05:59.254472265 (198h25m30.950321018s) [seg_pos={0 0}, upload=174871]: Segment reached optimal threshold of 13 pieces. Nov 7 18:05:59.254701491 (198h25m30.950550234s) [noise=true , address=45.91.116.14:4190 , piece_id=KFLJSQC26OTMY4MGM4GDC4TPODPPM2RQXCB2OPKFLXM7WF572FJQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:32854->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:32854->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32854->45.91.116.14:4190: use of closed network connection Nov 7 18:05:59.254711872 (198h25m30.950560665s) [noise=true , address=45.91.116.14:4190 , piece_id=KFLJSQC26OTMY4MGM4GDC4TPODPPM2RQXCB2OPKFLXM7WF572FJQ, seg_pos={0 0}, upload=174871]: Long tail cancellation due to optimal threshold Nov 7 18:05:59.254734474 (198h25m30.950583237s) [noise=true , address=77.90.31.11:4555 , piece_id=DMFSAYIRFEUC4YZFECB3RCMHCY4AY6H3PJOSWEY4X5N5FYRKWZRQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:36358->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:36358->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36358->77.90.31.11:4555: use of closed network connection Nov 7 18:05:59.254745345 (198h25m30.950594088s) [noise=true , address=77.90.31.11:4555 , piece_id=DMFSAYIRFEUC4YZFECB3RCMHCY4AY6H3PJOSWEY4X5N5FYRKWZRQ, seg_pos={0 0}, upload=174871]: Long tail cancellation due to optimal threshold Nov 7 18:05:59.254860735 (198h25m30.950709478s) [noise=true , address=77.90.31.16:4735 , piece_id=VEW2DC562VHY2L63V7RS6Y7GB343AX7WJNF4UK5A6ZIZOUBDYKJQ, seg_pos={0 0}, upload=174871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:33180->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:33180->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33180->77.90.31.16:4735: use of closed network connection Nov 7 18:05:59.254870764 (198h25m30.950719517s) [noise=true , address=77.90.31.16:4735 , piece_id=VEW2DC562VHY2L63V7RS6Y7GB343AX7WJNF4UK5A6ZIZOUBDYKJQ, seg_pos={0 0}, upload=174871]: Long tail cancellation due to optimal threshold Nov 7 18:05:59.255114588 (198h25m30.950963341s) [seg_pos={0 0}, upload=174871]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:05:59.255230920 (198h25m30.951079673s) [upload=174871]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:05:59.255238073 (198h25m30.951086826s) [upload=174871]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:05:59.567673149 (198h25m31.263521903s) [upload=174871]: Done uploading segments... Nov 7 18:05:59.567694030 (198h25m31.263542773s) [upload=174871]: Upload finished. err: <nil> Duration:* 30.004488653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.530149987s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:06:00 UTC) orbiter warning: context deadline exceeded Duration: 3.530166568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [......XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..X] duration: 30.002692981s trace: https://jaeger.dev.storj.io/trace/41e83e9a26e3e498 FAIL (2025-11-07 18:05:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: W3O4IGOKFBBZDWIYFF6XVZCL5OFBUWGMXR7AIAQPTZZUXUT4UA5Q) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: VTSLPHC4WORLUFTACO2TZNNSVDW56C5CTWRGARMPXD5ZV4JETMQQ) context deadline exceeded ======LOGS====== Nov 7 18:05:23.830580265 (198h24m55.526429019s) [upload=174859]: Starting upload Nov 7 18:05:23.830673794 (198h24m55.526522547s) [upload=174859]: Done starting upload Nov 7 18:05:23.830785777 (198h24m55.52663453s) [upload=174859]: Uploading segments... Nov 7 18:05:23.831202170 (198h24m55.527050943s) [upload=174859]: Got next segment. Inline: false Nov 7 18:05:23.831262946 (198h24m55.527111699s) [upload=174859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:05:23.831270380 (198h24m55.527119133s) [upload=174859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:05:24.080615046 (198h24m55.776463809s) [seg_pos={0 0}, upload=174859]: Begin upload segment... Nov 7 18:05:24.080704677 (198h24m55.77655343s) [seg_pos={0 0}, upload=174859]: Done begin upload segment. Nov 7 18:05:24.080788887 (198h24m55.77663764s) [noise=true , address=77.90.31.11:4555 , piece_id=H3MVEYYYTHSHF36BSZ56JHTD3BT4IPZEQE3UOCHG5IC6BF22FZ7Q, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.080836759 (198h24m55.776685512s) [noise=true , address=45.91.116.9:4262 , piece_id=Y3H3T4WHPD6SGOPVZIJXN3R5LVWGVZ7L5AZSCBRDV4MLLW24FYZA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.081433766 (198h24m55.777282509s) [noise=true , address=145.79.227.15:4874 , piece_id=CFEBH7B7K2SCADBHOROZYDIAUWOZKDMWECZWFL42XOA2HXZZTSKA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.081953966 (198h24m55.777802719s) [noise=true , address=145.79.227.11:4732 , piece_id=H7SSHZTPHCAITM7CMEISGEJRFTJGUWAK5HYF7Y6NRXMNGNFK2WUA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.082354348 (198h24m55.778203101s) [noise=true , address=145.79.227.30:4763 , piece_id=KGHFKGN3HTSGJEZNMLDU3KDJFPIJ6LPMNZZ4Y75TSVWWMUFHNOAA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.082777794 (198h24m55.778626547s) [noise=true , address=145.79.227.21:4474 , piece_id=D3CJSOUHJG62ZKWXAP6GKPSMXBNYKUFCN4SCU32VKY452EXSX4NQ, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.083303476 (198h24m55.779152229s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ3SFCZEC5GVHYS3QLLSIDH25MCEMHDL2PTRDVP25XTCYJCFEXHA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.083734266 (198h24m55.779583019s) [noise=true , address=77.90.31.10:4518 , piece_id=3GTMYYXLL6GA3STNORJ34ZBG2C2FW3JA6A5JJF5BQED63PAY4SRQ, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.084216665 (198h24m55.780065408s) [noise=true , address=45.91.116.11:4119 , piece_id=W3O4IGOKFBBZDWIYFF6XVZCL5OFBUWGMXR7AIAQPTZZUXUT4UA5Q, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.085461886 (198h24m55.781310639s) [noise=true , address=145.79.227.28:4725 , piece_id=HT7CLEO2GFJJG6GV2NRLNYVFBIP46AT4EBWAKVIFK4QVP26EUJUQ, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.087504856 (198h24m55.783353609s) [noise=true , address=145.79.227.27:4690 , piece_id=DRRC56R4Z5JSNXJL5VYN2U366RDCJXYLPFYZFYOV5Q2JNMCKTTBA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.089459128 (198h24m55.785307881s) [noise=true , address=145.79.227.22:4511 , piece_id=3VYMHG7XMK47DB5XUFBW4VHC7JDGFGE7O2XKL4IFUXFO5OQRXA5A, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.091594083 (198h24m55.787442836s) [noise=true , address=77.90.31.15:4698 , piece_id=VTSLPHC4WORLUFTACO2TZNNSVDW56C5CTWRGARMPXD5ZV4JETMQQ, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.094001948 (198h24m55.789850701s) [noise=true , address=45.91.116.10:4082 , piece_id=M27LG6INGVETS3BOX4GEZ456ATZT7O7ECYXJR3QYJX3Y2R3JV5NA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.096230010 (198h24m55.792078763s) [noise=true , address=145.79.227.27:4689 , piece_id=DYGR64JH2FICBWSM4VUPXBPBY7AWG5V37YHVG647RBUXO6MIX5OA, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:24.098797509 (198h24m55.794646272s) [noise=true , address=77.90.31.12:4589 , piece_id=PJLK5QRBTNOIRQQB6E4B5RHHFCW2JRC2NMP2REU2S3GOCCXOMU3Q, seg_pos={0 0}, upload=174859]: Uploading piece... Nov 7 18:05:25.780063170 (198h24m57.475911923s) [upload=174859]: Next returned nil segment Nov 7 18:05:25.780086795 (198h24m57.475935548s) [upload=174859]: Waiting for error group managing segments... Nov 7 18:05:26.063551915 (198h24m57.759400668s) [noise=true , address=145.79.227.11:4732 , piece_id=H7SSHZTPHCAITM7CMEISGEJRFTJGUWAK5HYF7Y6NRXMNGNFK2WUA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.068359770 (198h24m57.764208523s) [noise=true , address=145.79.227.22:4511 , piece_id=3VYMHG7XMK47DB5XUFBW4VHC7JDGFGE7O2XKL4IFUXFO5OQRXA5A, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.136884696 (198h24m57.832733449s) [noise=true , address=145.79.227.28:4725 , piece_id=HT7CLEO2GFJJG6GV2NRLNYVFBIP46AT4EBWAKVIFK4QVP26EUJUQ, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.167116164 (198h24m57.862964917s) [noise=true , address=145.79.227.27:4689 , piece_id=DYGR64JH2FICBWSM4VUPXBPBY7AWG5V37YHVG647RBUXO6MIX5OA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.204477859 (198h24m57.900326612s) [noise=true , address=145.79.227.15:4874 , piece_id=CFEBH7B7K2SCADBHOROZYDIAUWOZKDMWECZWFL42XOA2HXZZTSKA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.275055815 (198h24m57.970904558s) [noise=true , address=45.91.116.11:4119 , piece_id=W3O4IGOKFBBZDWIYFF6XVZCL5OFBUWGMXR7AIAQPTZZUXUT4UA5Q, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.319430815 (198h24m58.015279568s) [noise=true , address=45.91.116.9:4262 , piece_id=Y3H3T4WHPD6SGOPVZIJXN3R5LVWGVZ7L5AZSCBRDV4MLLW24FYZA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.358267598 (198h24m58.054116351s) [noise=true , address=45.91.116.10:4082 , piece_id=M27LG6INGVETS3BOX4GEZ456ATZT7O7ECYXJR3QYJX3Y2R3JV5NA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.375948296 (198h24m58.071797039s) [noise=true , address=145.79.227.21:4474 , piece_id=D3CJSOUHJG62ZKWXAP6GKPSMXBNYKUFCN4SCU32VKY452EXSX4NQ, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.376768258 (198h24m58.072617011s) [noise=true , address=145.79.227.30:4763 , piece_id=KGHFKGN3HTSGJEZNMLDU3KDJFPIJ6LPMNZZ4Y75TSVWWMUFHNOAA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.388356995 (198h24m58.084205748s) [noise=true , address=77.90.31.10:4518 , piece_id=3GTMYYXLL6GA3STNORJ34ZBG2C2FW3JA6A5JJF5BQED63PAY4SRQ, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.406971251 (198h24m58.102820004s) [noise=true , address=77.90.31.15:4698 , piece_id=VTSLPHC4WORLUFTACO2TZNNSVDW56C5CTWRGARMPXD5ZV4JETMQQ, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.471710588 (198h24m58.167559341s) [noise=true , address=145.79.227.27:4690 , piece_id=DRRC56R4Z5JSNXJL5VYN2U366RDCJXYLPFYZFYOV5Q2JNMCKTTBA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: <nil> Nov 7 18:05:26.471739644 (198h24m58.167588397s) [seg_pos={0 0}, upload=174859]: Segment reached optimal threshold of 13 pieces. Nov 7 18:05:26.471993999 (198h24m58.167842742s) [noise=true , address=77.90.31.11:4555 , piece_id=H3MVEYYYTHSHF36BSZ56JHTD3BT4IPZEQE3UOCHG5IC6BF22FZ7Q, seg_pos={0 0}, upload=174859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Nov 7 18:05:26.472005140 (198h24m58.167853893s) [noise=true , address=77.90.31.11:4555 , piece_id=H3MVEYYYTHSHF36BSZ56JHTD3BT4IPZEQE3UOCHG5IC6BF22FZ7Q, seg_pos={0 0}, upload=174859]: Long tail cancellation due to optimal threshold Nov 7 18:05:26.472265505 (198h24m58.168114268s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ3SFCZEC5GVHYS3QLLSIDH25MCEMHDL2PTRDVP25XTCYJCFEXHA, seg_pos={0 0}, upload=174859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Nov 7 18:05:26.472287868 (198h24m58.168136631s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ3SFCZEC5GVHYS3QLLSIDH25MCEMHDL2PTRDVP25XTCYJCFEXHA, seg_pos={0 0}, upload=174859]: Long tail cancellation due to optimal threshold Nov 7 18:05:26.472337312 (198h24m58.168186065s) [noise=true , address=77.90.31.12:4589 , piece_id=PJLK5QRBTNOIRQQB6E4B5RHHFCW2JRC2NMP2REU2S3GOCCXOMU3Q, seg_pos={0 0}, upload=174859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: rpc: dial tcp 77.90.31.12:4589: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Nov 7 18:05:26.472348674 (198h24m58.168197427s) [noise=true , address=77.90.31.12:4589 , piece_id=PJLK5QRBTNOIRQQB6E4B5RHHFCW2JRC2NMP2REU2S3GOCCXOMU3Q, seg_pos={0 0}, upload=174859]: Long tail cancellation due to optimal threshold Nov 7 18:05:26.472373310 (198h24m58.168222063s) [seg_pos={0 0}, upload=174859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:05:26.472462851 (198h24m58.168311604s) [upload=174859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:05:26.472470996 (198h24m58.168319749s) [upload=174859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:05:26.771859597 (198h24m58.46770836s) [upload=174859]: Done uploading segments... Nov 7 18:05:26.771886890 (198h24m58.467735643s) [upload=174859]: Upload finished. err: <nil> Duration: 30.002738539s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.463520887s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:05:29 UTC) orbiter warning: context deadline exceeded Duration: 3.463551175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..] duration: 18.540337479s trace: https://jaeger.dev.storj.io/trace/7cbfc9f543672cc1 FAIL (2025-11-07 18:04:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ) context deadline exceeded ======LOGS====== Nov 7 18:02:25.980443327 (198h21m57.67629208s) [upload=174819]: Starting upload Nov 7 18:02:25.980502540 (198h21m57.676351283s) [upload=174819]: Done starting upload Nov 7 18:02:25.980708362 (198h21m57.676557115s) [upload=174819]: Uploading segments... Nov 7 18:02:25.980911890 (198h21m57.676760643s) [upload=174819]: Got next segment. Inline: false Nov 7 18:02:25.981016359 (198h21m57.676865112s) [upload=174819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:02:25.981024855 (198h21m57.676873608s) [upload=174819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:02:26.239397983 (198h21m57.935246736s) [seg_pos={0 0}, upload=174819]: Begin upload segment... Nov 7 18:02:26.239464600 (198h21m57.935313353s) [seg_pos={0 0}, upload=174819]: Done begin upload segment. Nov 7 18:02:26.239513955 (198h21m57.935362708s) [noise=true , address=77.90.31.10:4517 , piece_id=X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.240771459 (198h21m57.936620212s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.241411448 (198h21m57.937260201s) [noise=true , address=145.79.227.15:4876 , piece_id=2RX2PHBEHMFPXFEE6GLNYIWIHSASZ7LNPF6ITHERU7KDD6ZOH2CQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.242028543 (198h21m57.937877336s) [noise=true , address=45.91.116.13:4154 , piece_id=73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.242579612 (198h21m57.938428385s) [noise=true , address=77.90.31.17:4770 , piece_id=3JVLE5VWSUFOGHBXFBYFUHYZHETKKZY3AGYJNGNTADPZD4TNBJ3A, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.243270688 (198h21m57.939119442s) [noise=true , address=145.79.227.31:4797 , piece_id=XNBIFOZZGE5EDVAVQ3FDGMF6DJV76RMJTZBTRH5273CWRKO7ASFQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.243840764 (198h21m57.939689517s) [noise=true , address=145.79.227.8:4329 , piece_id=CHZ5RCAXZD457JVVOMLEO62UMZBQWYMZ5O7ARISCJTFIOGAXPNKA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.244391993 (198h21m57.940240777s) [noise=true , address=45.91.116.14:4190 , piece_id=YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.244940137 (198h21m57.94078889s) [noise=true , address=145.79.227.17:4970 , piece_id=P3EG7JRRZGEL6T4O6UZ6Z4ILXGWRXTWAQHA4CAKUZ7H6P2EFZ7XA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.248886421 (198h21m57.944735175s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.251505870 (198h21m57.947354623s) [noise=true , address=145.79.227.24:4581 , piece_id=WGPHWKF7IR6P2Q5EEV6X54TSUVD4IEJSFDG5VXD7TIV37ZOHLPSA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.253765523 (198h21m57.949614276s) [noise=true , address=145.79.227.13:4802 , piece_id=ZJZ2ZBRRTZNXVJLPKFKFFQWZKYV3JX5ABBX2ZT26PWFASNOREMCQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.256253801 (198h21m57.952102584s) [noise=true , address=45.91.116.6:4974 , piece_id=TPOGBQAKKVKR6LZA2PBUKCW2BFUA4CUGEMN7RO42QY5CTRL4WQ4A, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.260548669 (198h21m57.956397452s) [noise=true , address=145.79.227.26:4655 , piece_id=XRBMET3AF5ZQQ7UIXOKC7LRUE2FSX26T76IGFCLCX5M22PPFCTMQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.266266477 (198h21m57.96211527s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.268616682 (198h21m57.964465436s) [noise=true , address=77.90.31.19:4843 , piece_id=D67UJ7JI4E2NWNJ2HRI7M7MBMWQNRTJTRA7RTXRRUKQW7EQBSYAQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:27.930472071 (198h21m59.626320824s) [upload=174819]: Next returned nil segment Nov 7 18:02:27.930487160 (198h21m59.626335914s) [upload=174819]: Waiting for error group managing segments... Nov 7 18:02:28.345299434 (198h22m0.041148187s) [noise=true , address=145.79.227.15:4876 , piece_id=2RX2PHBEHMFPXFEE6GLNYIWIHSASZ7LNPF6ITHERU7KDD6ZOH2CQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.360520599 (198h22m0.056369352s) [noise=true , address=145.79.227.17:4970 , piece_id=P3EG7JRRZGEL6T4O6UZ6Z4ILXGWRXTWAQHA4CAKUZ7H6P2EFZ7XA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.374274499 (198h22m0.070123252s) [noise=true , address=145.79.227.13:4802 , piece_id=ZJZ2ZBRRTZNXVJLPKFKFFQWZKYV3JX5ABBX2ZT26PWFASNOREMCQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.443146085 (198h22m0.138994829s) [noise=true , address=145.79.227.31:4797 , piece_id=XNBIFOZZGE5EDVAVQ3FDGMF6DJV76RMJTZBTRH5273CWRKO7ASFQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.525765280 (198h22m0.221614023s) [noise=true , address=145.79.227.8:4329 , piece_id=CHZ5RCAXZD457JVVOMLEO62UMZBQWYMZ5O7ARISCJTFIOGAXPNKA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.528056223 (198h22m0.223904976s) [noise=true , address=77.90.31.10:4517 , piece_id=X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.535857990 (198h22m0.231706743s) [noise=true , address=45.91.116.6:4974 , piece_id=TPOGBQAKKVKR6LZA2PBUKCW2BFUA4CUGEMN7RO42QY5CTRL4WQ4A, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.550156727 (198h22m0.24600548s) [noise=true , address=77.90.31.17:4770 , piece_id=3JVLE5VWSUFOGHBXFBYFUHYZHETKKZY3AGYJNGNTADPZD4TNBJ3A, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.567089261 (198h22m0.262938014s) [noise=true , address=45.91.116.13:4154 , piece_id=73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.599614876 (198h22m0.295463629s) [noise=true , address=145.79.227.26:4655 , piece_id=XRBMET3AF5ZQQ7UIXOKC7LRUE2FSX26T76IGFCLCX5M22PPFCTMQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.606810719 (198h22m0.302659472s) [noise=true , address=45.91.116.14:4190 , piece_id=YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.651429463 (198h22m0.347278216s) [noise=true , address=145.79.227.24:4581 , piece_id=WGPHWKF7IR6P2Q5EEV6X54TSUVD4IEJSFDG5VXD7TIV37ZOHLPSA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.673567079 (198h22m0.369415832s) [noise=true , address=77.90.31.19:4843 , piece_id=D67UJ7JI4E2NWNJ2HRI7M7MBMWQNRTJTRA7RTXRRUKQW7EQBSYAQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.673588509 (198h22m0.369437252s) [seg_pos={0 0}, upload=174819]: Segment reached optimal threshold of 13 pieces. Nov 7 18:02:28.673845940 (198h22m0.369694693s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Nov 7 18:02:28.673854015 (198h22m0.369702768s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.673927274 (198h22m0.369776017s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Nov 7 18:02:28.673932243 (198h22m0.369780996s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.674242274 (198h22m0.370091017s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: context canceled; context canceled Nov 7 18:02:28.674254797 (198h22m0.37010355s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.674278282 (198h22m0.370127035s) [seg_pos={0 0}, upload=174819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:02:28.674396758 (198h22m0.370245511s) [upload=174819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:02:28.674405285 (198h22m0.370254038s) [upload=174819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:02:28.979445207 (198h22m0.67529397s) [upload=174819]: Done uploading segments... Nov 7 18:02:28.979474353 (198h22m0.675323096s) [upload=174819]: Upload finished. err: <nil> Duration: 18.540368998s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.857150957s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:04:16 UTC) orbiter warning: context deadline exceeded Duration: 3.857189701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.757284919s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:03:33 UTC) orbiter warning: context deadline exceeded Duration: 3.757314324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [........XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X.] duration: 14.485603727s trace: https://jaeger.dev.storj.io/trace/7cbfc9f543672cc1 FAIL (2025-11-07 18:03:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ) context deadline exceeded ======LOGS====== Nov 7 18:02:25.980443327 (198h21m57.67629208s) [upload=174819]: Starting upload Nov 7 18:02:25.980502540 (198h21m57.676351283s) [upload=174819]: Done starting upload Nov 7 18:02:25.980708362 (198h21m57.676557115s) [upload=174819]: Uploading segments... Nov 7 18:02:25.980911890 (198h21m57.676760643s) [upload=174819]: Got next segment. Inline: false Nov 7 18:02:25.981016359 (198h21m57.676865112s) [upload=174819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:02:25.981024855 (198h21m57.676873608s) [upload=174819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:02:26.239397983 (198h21m57.935246736s) [seg_pos={0 0}, upload=174819]: Begin upload segment... Nov 7 18:02:26.239464600 (198h21m57.935313353s) [seg_pos={0 0}, upload=174819]: Done begin upload segment. Nov 7 18:02:26.239513955 (198h21m57.935362708s) [noise=true , address=77.90.31.10:4517 , piece_id=X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.240771459 (198h21m57.936620212s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.241411448 (198h21m57.937260201s) [noise=true , address=145.79.227.15:4876 , piece_id=2RX2PHBEHMFPXFEE6GLNYIWIHSASZ7LNPF6ITHERU7KDD6ZOH2CQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.242028543 (198h21m57.937877336s) [noise=true , address=45.91.116.13:4154 , piece_id=73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.242579612 (198h21m57.938428385s) [noise=true , address=77.90.31.17:4770 , piece_id=3JVLE5VWSUFOGHBXFBYFUHYZHETKKZY3AGYJNGNTADPZD4TNBJ3A, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.243270688 (198h21m57.939119442s) [noise=true , address=145.79.227.31:4797 , piece_id=XNBIFOZZGE5EDVAVQ3FDGMF6DJV76RMJTZBTRH5273CWRKO7ASFQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.243840764 (198h21m57.939689517s) [noise=true , address=145.79.227.8:4329 , piece_id=CHZ5RCAXZD457JVVOMLEO62UMZBQWYMZ5O7ARISCJTFIOGAXPNKA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.244391993 (198h21m57.940240777s) [noise=true , address=45.91.116.14:4190 , piece_id=YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.244940137 (198h21m57.94078889s) [noise=true , address=145.79.227.17:4970 , piece_id=P3EG7JRRZGEL6T4O6UZ6Z4ILXGWRXTWAQHA4CAKUZ7H6P2EFZ7XA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.248886421 (198h21m57.944735175s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.251505870 (198h21m57.947354623s) [noise=true , address=145.79.227.24:4581 , piece_id=WGPHWKF7IR6P2Q5EEV6X54TSUVD4IEJSFDG5VXD7TIV37ZOHLPSA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.253765523 (198h21m57.949614276s) [noise=true , address=145.79.227.13:4802 , piece_id=ZJZ2ZBRRTZNXVJLPKFKFFQWZKYV3JX5ABBX2ZT26PWFASNOREMCQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.256253801 (198h21m57.952102584s) [noise=true , address=45.91.116.6:4974 , piece_id=TPOGBQAKKVKR6LZA2PBUKCW2BFUA4CUGEMN7RO42QY5CTRL4WQ4A, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.260548669 (198h21m57.956397452s) [noise=true , address=145.79.227.26:4655 , piece_id=XRBMET3AF5ZQQ7UIXOKC7LRUE2FSX26T76IGFCLCX5M22PPFCTMQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.266266477 (198h21m57.96211527s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.268616682 (198h21m57.964465436s) [noise=true , address=77.90.31.19:4843 , piece_id=D67UJ7JI4E2NWNJ2HRI7M7MBMWQNRTJTRA7RTXRRUKQW7EQBSYAQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:27.930472071 (198h21m59.626320824s) [upload=174819]: Next returned nil segment Nov 7 18:02:27.930487160 (198h21m59.626335914s) [upload=174819]: Waiting for error group managing segments... Nov 7 18:02:28.345299434 (198h22m0.041148187s) [noise=true , address=145.79.227.15:4876 , piece_id=2RX2PHBEHMFPXFEE6GLNYIWIHSASZ7LNPF6ITHERU7KDD6ZOH2CQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.360520599 (198h22m0.056369352s) [noise=true , address=145.79.227.17:4970 , piece_id=P3EG7JRRZGEL6T4O6UZ6Z4ILXGWRXTWAQHA4CAKUZ7H6P2EFZ7XA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.374274499 (198h22m0.070123252s) [noise=true , address=145.79.227.13:4802 , piece_id=ZJZ2ZBRRTZNXVJLPKFKFFQWZKYV3JX5ABBX2ZT26PWFASNOREMCQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.443146085 (198h22m0.138994829s) [noise=true , address=145.79.227.31:4797 , piece_id=XNBIFOZZGE5EDVAVQ3FDGMF6DJV76RMJTZBTRH5273CWRKO7ASFQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.525765280 (198h22m0.221614023s) [noise=true , address=145.79.227.8:4329 , piece_id=CHZ5RCAXZD457JVVOMLEO62UMZBQWYMZ5O7ARISCJTFIOGAXPNKA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.528056223 (198h22m0.223904976s) [noise=true , address=77.90.31.10:4517 , piece_id=X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.535857990 (198h22m0.231706743s) [noise=true , address=45.91.116.6:4974 , piece_id=TPOGBQAKKVKR6LZA2PBUKCW2BFUA4CUGEMN7RO42QY5CTRL4WQ4A, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.550156727 (198h22m0.24600548s) [noise=true , address=77.90.31.17:4770 , piece_id=3JVLE5VWSUFOGHBXFBYFUHYZHETKKZY3AGYJNGNTADPZD4TNBJ3A, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.567089261 (198h22m0.262938014s) [noise=true , address=45.91.116.13:4154 , piece_id=73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.599614876 (198h22m0.295463629s) [noise=true , address=145.79.227.26:4655 , piece_id=XRBMET3AF5ZQQ7UIXOKC7LRUE2FSX26T76IGFCLCX5M22PPFCTMQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.606810719 (198h22m0.302659472s) [noise=true , address=45.91.116.14:4190 , piece_id=YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.651429463 (198h22m0.347278216s) [noise=true , address=145.79.227.24:4581 , piece_id=WGPHWKF7IR6P2Q5EEV6X54TSUVD4IEJSFDG5VXD7TIV37ZOHLPSA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.673567079 (198h22m0.369415832s) [noise=true , address=77.90.31.19:4843 , piece_id=D67UJ7JI4E2NWNJ2HRI7M7MBMWQNRTJTRA7RTXRRUKQW7EQBSYAQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.673588509 (198h22m0.369437252s) [seg_pos={0 0}, upload=174819]: Segment reached optimal threshold of 13 pieces. Nov 7 18:02:28.673845940 (198h22m0.369694693s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Nov 7 18:02:28.673854015 (198h22m0.369702768s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.673927274 (198h22m0.369776017s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Nov 7 18:02:28.673932243 (198h22m0.369780996s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.674242274 (198h22m0.370091017s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: context canceled; context canceled Nov 7 18:02:28.674254797 (198h22m0.37010355s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.674278282 (198h22m0.370127035s) [seg_pos={0 0}, upload=174819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:02:28.674396758 (198h22m0.370245511s) [upload=174819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:02:28.674405285 (198h22m0.370254038s) [upload=174819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:02:28.979445207 (198h22m0.67529397s) [upload=174819]: Done uploading segments... Nov 7 18:02:28.979474353 (198h22m0.675323096s) [upload=174819]: Upload finished. err: <nil> Duration: 14.485636338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.........XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X] duration: 30.002911901s trace: https://jaeger.dev.storj.io/trace/7cbfc9f543672cc1 FAIL (2025-11-07 18:02:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ) context deadline exceeded ======LOGS====== Nov 7 18:02:25.980443327 (198h21m57.67629208s) [upload=174819]: Starting upload Nov 7 18:02:25.980502540 (198h21m57.676351283s) [upload=174819]: Done starting upload Nov 7 18:02:25.980708362 (198h21m57.676557115s) [upload=174819]: Uploading segments... Nov 7 18:02:25.980911890 (198h21m57.676760643s) [upload=174819]: Got next segment. Inline: false Nov 7 18:02:25.981016359 (198h21m57.676865112s) [upload=174819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:02:25.981024855 (198h21m57.676873608s) [upload=174819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:02:26.239397983 (198h21m57.935246736s) [seg_pos={0 0}, upload=174819]: Begin upload segment... Nov 7 18:02:26.239464600 (198h21m57.935313353s) [seg_pos={0 0}, upload=174819]: Done begin upload segment. Nov 7 18:02:26.239513955 (198h21m57.935362708s) [noise=true , address=77.90.31.10:4517 , piece_id=X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.240771459 (198h21m57.936620212s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.241411448 (198h21m57.937260201s) [noise=true , address=145.79.227.15:4876 , piece_id=2RX2PHBEHMFPXFEE6GLNYIWIHSASZ7LNPF6ITHERU7KDD6ZOH2CQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.242028543 (198h21m57.937877336s) [noise=true , address=45.91.116.13:4154 , piece_id=73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.242579612 (198h21m57.938428385s) [noise=true , address=77.90.31.17:4770 , piece_id=3JVLE5VWSUFOGHBXFBYFUHYZHETKKZY3AGYJNGNTADPZD4TNBJ3A, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.243270688 (198h21m57.939119442s) [noise=true , address=145.79.227.31:4797 , piece_id=XNBIFOZZGE5EDVAVQ3FDGMF6DJV76RMJTZBTRH5273CWRKO7ASFQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.243840764 (198h21m57.939689517s) [noise=true , address=145.79.227.8:4329 , piece_id=CHZ5RCAXZD457JVVOMLEO62UMZBQWYMZ5O7ARISCJTFIOGAXPNKA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.244391993 (198h21m57.940240777s) [noise=true , address=45.91.116.14:4190 , piece_id=YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.244940137 (198h21m57.94078889s) [noise=true , address=145.79.227.17:4970 , piece_id=P3EG7JRRZGEL6T4O6UZ6Z4ILXGWRXTWAQHA4CAKUZ7H6P2EFZ7XA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.248886421 (198h21m57.944735175s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.251505870 (198h21m57.947354623s) [noise=true , address=145.79.227.24:4581 , piece_id=WGPHWKF7IR6P2Q5EEV6X54TSUVD4IEJSFDG5VXD7TIV37ZOHLPSA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.253765523 (198h21m57.949614276s) [noise=true , address=145.79.227.13:4802 , piece_id=ZJZ2ZBRRTZNXVJLPKFKFFQWZKYV3JX5ABBX2ZT26PWFASNOREMCQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.256253801 (198h21m57.952102584s) [noise=true , address=45.91.116.6:4974 , piece_id=TPOGBQAKKVKR6LZA2PBUKCW2BFUA4CUGEMN7RO42QY5CTRL4WQ4A, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.260548669 (198h21m57.956397452s) [noise=true , address=145.79.227.26:4655 , piece_id=XRBMET3AF5ZQQ7UIXOKC7LRUE2FSX26T76IGFCLCX5M22PPFCTMQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.266266477 (198h21m57.96211527s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:26.268616682 (198h21m57.964465436s) [noise=true , address=77.90.31.19:4843 , piece_id=D67UJ7JI4E2NWNJ2HRI7M7MBMWQNRTJTRA7RTXRRUKQW7EQBSYAQ, seg_pos={0 0}, upload=174819]: Uploading piece... Nov 7 18:02:27.930472071 (198h21m59.626320824s) [upload=174819]: Next returned nil segment Nov 7 18:02:27.930487160 (198h21m59.626335914s) [upload=174819]: Waiting for error group managing segments... Nov 7 18:02:28.345299434 (198h22m0.041148187s) [noise=true , address=145.79.227.15:4876 , piece_id=2RX2PHBEHMFPXFEE6GLNYIWIHSASZ7LNPF6ITHERU7KDD6ZOH2CQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.360520599 (198h22m0.056369352s) [noise=true , address=145.79.227.17:4970 , piece_id=P3EG7JRRZGEL6T4O6UZ6Z4ILXGWRXTWAQHA4CAKUZ7H6P2EFZ7XA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.374274499 (198h22m0.070123252s) [noise=true , address=145.79.227.13:4802 , piece_id=ZJZ2ZBRRTZNXVJLPKFKFFQWZKYV3JX5ABBX2ZT26PWFASNOREMCQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.443146085 (198h22m0.138994829s) [noise=true , address=145.79.227.31:4797 , piece_id=XNBIFOZZGE5EDVAVQ3FDGMF6DJV76RMJTZBTRH5273CWRKO7ASFQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.525765280 (198h22m0.221614023s) [noise=true , address=145.79.227.8:4329 , piece_id=CHZ5RCAXZD457JVVOMLEO62UMZBQWYMZ5O7ARISCJTFIOGAXPNKA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.528056223 (198h22m0.223904976s) [noise=true , address=77.90.31.10:4517 , piece_id=X3AWQOBZ2UVJYJ5WPNI24BFGXLOQUOENCJULBSRVZBSWXRQ54JGA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.535857990 (198h22m0.231706743s) [noise=true , address=45.91.116.6:4974 , piece_id=TPOGBQAKKVKR6LZA2PBUKCW2BFUA4CUGEMN7RO42QY5CTRL4WQ4A, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.550156727 (198h22m0.24600548s) [noise=true , address=77.90.31.17:4770 , piece_id=3JVLE5VWSUFOGHBXFBYFUHYZHETKKZY3AGYJNGNTADPZD4TNBJ3A, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.567089261 (198h22m0.262938014s) [noise=true , address=45.91.116.13:4154 , piece_id=73P77X3LMLPKT4KB7IZFDY7SMK3UXVWOO6FQPJKN5VFMBQLLT7NQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.599614876 (198h22m0.295463629s) [noise=true , address=145.79.227.26:4655 , piece_id=XRBMET3AF5ZQQ7UIXOKC7LRUE2FSX26T76IGFCLCX5M22PPFCTMQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.606810719 (198h22m0.302659472s) [noise=true , address=45.91.116.14:4190 , piece_id=YMIFDCZ6TB3DITZGIP3MY42HDCSSHG67I6WAUV5F4VFNWY63NXMQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.651429463 (198h22m0.347278216s) [noise=true , address=145.79.227.24:4581 , piece_id=WGPHWKF7IR6P2Q5EEV6X54TSUVD4IEJSFDG5VXD7TIV37ZOHLPSA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.673567079 (198h22m0.369415832s) [noise=true , address=77.90.31.19:4843 , piece_id=D67UJ7JI4E2NWNJ2HRI7M7MBMWQNRTJTRA7RTXRRUKQW7EQBSYAQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: <nil> Nov 7 18:02:28.673588509 (198h22m0.369437252s) [seg_pos={0 0}, upload=174819]: Segment reached optimal threshold of 13 pieces. Nov 7 18:02:28.673845940 (198h22m0.369694693s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Nov 7 18:02:28.673854015 (198h22m0.369702768s) [noise=true , address=45.91.116.11:4119 , piece_id=6F66QYGVSLMT2OP4U5ESQBZOY5UKR3HLB2HA4QY27DGVCB4CPHXA, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.673927274 (198h22m0.369776017s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Nov 7 18:02:28.673932243 (198h22m0.369780996s) [noise=true , address=77.90.31.13:4628 , piece_id=U7KPWFVXFITZGSW4HWMIMP2ICX43W2AHH7KBOVP66SG7TJEPWTCA, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.674242274 (198h22m0.370091017s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: context canceled; context canceled Nov 7 18:02:28.674254797 (198h22m0.37010355s) [noise=true , address=77.90.31.21:4879 , piece_id=FUPCKL5T3I6UK6K5Z4I2BTMN2UUY3EXWIVB7TDVIWLFPSZHDLIYQ, seg_pos={0 0}, upload=174819]: Long tail cancellation due to optimal threshold Nov 7 18:02:28.674278282 (198h22m0.370127035s) [seg_pos={0 0}, upload=174819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:02:28.674396758 (198h22m0.370245511s) [upload=174819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:02:28.674405285 (198h22m0.370254038s) [upload=174819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:02:28.979445207 (198h22m0.67529397s) [upload=174819]: Done uploading segments... Nov 7 18:02:28.979474353 (198h22m0.675323096s) [upload=174819]: Upload finished. err: <nil> Duration: 30.002944564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.553681101s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:02:42 UTC) orbiter warning: context deadline exceeded Duration: 3.553711479s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.578219235s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:02:20 UTC) orbiter warning: context deadline exceeded Duration: 3.578247509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [..........XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.] duration: 24.752419012s trace: https://jaeger.dev.storj.io/trace/69cc3cc02350ba23 FAIL (2025-11-07 18:01:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: 5HEE5776LOPMUBWY23P3NMIOQ6GXZHDGLZR2WIEUPZDG6SCNJWHQ) context deadline exceeded; piecestore: (Node ID: 18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, Piece ID: UQ2DVP2OCJBXP237XF3PEIUG5TLRMJ7HHVL6DZJLO2VQDWW5KYIQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: HXWDF5J6QKCBRWMW47DKSEI7TE5BSISOHNRBQV3WQH74ZZRQWH2A) context deadline exceeded ======LOGS====== Nov 7 18:00:38.456604225 (198h20m10.152453018s) [upload=174795]: Starting upload Nov 7 18:00:38.456697002 (198h20m10.152545795s) [upload=174795]: Done starting upload Nov 7 18:00:38.456746345 (198h20m10.152595108s) [upload=174795]: Uploading segments... Nov 7 18:00:38.457038933 (198h20m10.152887686s) [upload=174795]: Got next segment. Inline: false Nov 7 18:00:38.457129084 (198h20m10.152977827s) [upload=174795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:00:38.457138502 (198h20m10.152987255s) [upload=174795]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:00:38.747435703 (198h20m10.443284456s) [seg_pos={0 0}, upload=174795]: Begin upload segment... Nov 7 18:00:38.747501588 (198h20m10.443350341s) [seg_pos={0 0}, upload=174795]: Done begin upload segment. Nov 7 18:00:38.747549399 (198h20m10.443398152s) [noise=true , address=145.79.227.22:4510 , piece_id=X3IWHVTRRLDREQYN2HH64URYRSBCZSKHP453DVTN62IDMTMPSTOQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.751832135 (198h20m10.447680888s) [noise=true , address=45.91.116.9:4263 , piece_id=HXWDF5J6QKCBRWMW47DKSEI7TE5BSISOHNRBQV3WQH74ZZRQWH2A, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.753088377 (198h20m10.44893714s) [noise=true , address=45.91.116.14:4190 , piece_id=JF3IIN4AMZSKMROWP5PDJR5UJDFCOLMEYC3NZBS4R4WVNYO6H4QA, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.753933676 (198h20m10.449782429s) [noise=true , address=45.91.116.8:4048 , piece_id=XWLMXVUY3RV2KJS7M2V2IAB7XNYWOCMSMURGGNU33ZEXIRM2ZYLA, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.754526735 (198h20m10.450375488s) [noise=true , address=145.79.227.30:4761 , piece_id=5DO2RD74OX4PT5LNN2IPQOV6LT6PBA3PYMC3IODXFWSETAUAKUOQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.755097332 (198h20m10.450946086s) [noise=true , address=145.79.227.32:4836 , piece_id=TTXQIIYFDHFS43UQH5LBZONHX7YJLXRS6DSAJ7SRQW3DD46A2P7A, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.755623805 (198h20m10.451472568s) [noise=true , address=145.79.227.28:4725 , piece_id=6XS2BOMMKE7TADHNPT5ZGTRUUGSAYSY7BBQMHIIUFDX6KI5Q3DWA, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.756248655 (198h20m10.452097408s) [noise=true , address=45.91.116.7:4010 , piece_id=K73CC7XUA2AHMXRADZEWBSVHY5IXXYXEXEJI4VWV76OQ2VTVYOEQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.756815013 (198h20m10.452663766s) [noise=true , address=77.90.31.21:4879 , piece_id=4CLWXXBVVSAVKUPM72Q5P2OLVPWIDVOEMM2GAQO6OXHJSURL7DTQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.758837093 (198h20m10.454685856s) [noise=true , address=145.79.227.25:4617 , piece_id=MWZNA4FOK4YHJRGPDKJVE6A5QVRQARIVJD2SAAX2B5MQLOPJBU5A, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.759687011 (198h20m10.455535764s) [noise=true , address=145.79.227.15:4876 , piece_id=GN5GEGC6FPXUTDLN6XFY5CPQMIGRPTDHIO7MLROBNVMGSKPZA3IQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.760284570 (198h20m10.456133313s) [noise=true , address=77.90.31.15:4698 , piece_id=5HEE5776LOPMUBWY23P3NMIOQ6GXZHDGLZR2WIEUPZDG6SCNJWHQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.760973431 (198h20m10.456822194s) [noise=true , address=145.79.227.13:4803 , piece_id=KW3A7RGL77V3I4IR7L36GFOHWIWQZNTPPSDUPIHQ7K4JMPGBFUMQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.761679346 (198h20m10.457528099s) [noise=true , address=145.79.227.10:4403 , piece_id=ELMFQMWILFX5NNE4ECLT6QPN3GE4VW2ZQMFEHVF4QBVEU3P6JBDQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.763257020 (198h20m10.459105773s) [noise=true , address=145.79.227.26:4655 , piece_id=5X6ABUUXWAPIKTOAVLTZRZRJYTHTYTG36SRJ5NEHVQ6NVX5FEDUQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.764773147 (198h20m10.46062191s) [noise=true , address=77.90.31.17:4772 , piece_id=UQ2DVP2OCJBXP237XF3PEIUG5TLRMJ7HHVL6DZJLO2VQDWW5KYIQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:40.560729193 (198h20m12.256577936s) [upload=174795]: Next returned nil segment Nov 7 18:00:40.560743740 (198h20m12.256592493s) [upload=174795]: Waiting for error group managing segments... Nov 7 18:00:40.863633679 (198h20m12.559482442s) [noise=true , address=145.79.227.32:4836 , piece_id=TTXQIIYFDHFS43UQH5LBZONHX7YJLXRS6DSAJ7SRQW3DD46A2P7A, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:40.940082104 (198h20m12.635930857s) [noise=true , address=145.79.227.26:4655 , piece_id=5X6ABUUXWAPIKTOAVLTZRZRJYTHTYTG36SRJ5NEHVQ6NVX5FEDUQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:40.983226180 (198h20m12.679074933s) [noise=true , address=145.79.227.10:4403 , piece_id=ELMFQMWILFX5NNE4ECLT6QPN3GE4VW2ZQMFEHVF4QBVEU3P6JBDQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.034203803 (198h20m12.730052556s) [noise=true , address=145.79.227.25:4617 , piece_id=MWZNA4FOK4YHJRGPDKJVE6A5QVRQARIVJD2SAAX2B5MQLOPJBU5A, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.053008571 (198h20m12.748857324s) [noise=true , address=145.79.227.30:4761 , piece_id=5DO2RD74OX4PT5LNN2IPQOV6LT6PBA3PYMC3IODXFWSETAUAKUOQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.070400158 (198h20m12.766248921s) [noise=true , address=145.79.227.13:4803 , piece_id=KW3A7RGL77V3I4IR7L36GFOHWIWQZNTPPSDUPIHQ7K4JMPGBFUMQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.415706468 (198h20m13.111555221s) [noise=true , address=77.90.31.15:4698 , piece_id=5HEE5776LOPMUBWY23P3NMIOQ6GXZHDGLZR2WIEUPZDG6SCNJWHQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.417404581 (198h20m13.113253334s) [noise=true , address=145.79.227.15:4876 , piece_id=GN5GEGC6FPXUTDLN6XFY5CPQMIGRPTDHIO7MLROBNVMGSKPZA3IQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.469060125 (198h20m13.164908878s) [noise=true , address=145.79.227.28:4725 , piece_id=6XS2BOMMKE7TADHNPT5ZGTRUUGSAYSY7BBQMHIIUFDX6KI5Q3DWA, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.498378913 (198h20m13.194227666s) [noise=true , address=77.90.31.17:4772 , piece_id=UQ2DVP2OCJBXP237XF3PEIUG5TLRMJ7HHVL6DZJLO2VQDWW5KYIQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.731346175 (198h20m13.427194928s) [noise=true , address=145.79.227.22:4510 , piece_id=X3IWHVTRRLDREQYN2HH64URYRSBCZSKHP453DVTN62IDMTMPSTOQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.845993869 (198h20m13.541842622s) [noise=true , address=45.91.116.9:4263 , piece_id=HXWDF5J6QKCBRWMW47DKSEI7TE5BSISOHNRBQV3WQH74ZZRQWH2A, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.872505804 (198h20m13.568354567s) [noise=true , address=45.91.116.14:4190 , piece_id=JF3IIN4AMZSKMROWP5PDJR5UJDFCOLMEYC3NZBS4R4WVNYO6H4QA, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.872553955 (198h20m13.568402708s) [seg_pos={0 0}, upload=174795]: Segment reached optimal threshold of 13 pieces. Nov 7 18:00:41.872825513 (198h20m13.568674276s) [noise=true , address=77.90.31.21:4879 , piece_id=4CLWXXBVVSAVKUPM72Q5P2OLVPWIDVOEMM2GAQO6OXHJSURL7DTQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 188.166.225.170:40814->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:40814->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40814->77.90.31.21:4879: use of closed network connection Nov 7 18:00:41.872835481 (198h20m13.568684234s) [noise=true , address=77.90.31.21:4879 , piece_id=4CLWXXBVVSAVKUPM72Q5P2OLVPWIDVOEMM2GAQO6OXHJSURL7DTQ, seg_pos={0 0}, upload=174795]: Long tail cancellation due to optimal threshold Nov 7 18:00:41.872913841 (198h20m13.568762604s) [noise=true , address=45.91.116.8:4048 , piece_id=XWLMXVUY3RV2KJS7M2V2IAB7XNYWOCMSMURGGNU33ZEXIRM2ZYLA, seg_pos={0 0}, upload=174795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:41.872947094 (198h20m13.568795848s) [noise=true , address=45.91.116.8:4048 , piece_id=XWLMXVUY3RV2KJS7M2V2IAB7XNYWOCMSMURGGNU33ZEXIRM2ZYLA, seg_pos={0 0}, upload=174795]: Long tail cancellation due to optimal threshold Nov 7 18:00:41.873095577 (198h20m13.56894434s) [noise=true , address=45.91.116.7:4010 , piece_id=K73CC7XUA2AHMXRADZEWBSVHY5IXXYXEXEJI4VWV76OQ2VTVYOEQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:41.873107099 (198h20m13.568955852s) [noise=true , address=45.91.116.7:4010 , piece_id=K73CC7XUA2AHMXRADZEWBSVHY5IXXYXEXEJI4VWV76OQ2VTVYOEQ, seg_pos={0 0}, upload=174795]: Long tail cancellation due to optimal threshold Nov 7 18:00:41.873132047 (198h20m13.5689808s) [seg_pos={0 0}, upload=174795]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:00:41.873226467 (198h20m13.56907522s) [upload=174795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:00:41.873234792 (198h20m13.569083545s) [upload=174795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:00:42.203462148 (198h20m13.899310911s) [upload=174795]: Done uploading segments... Nov 7 18:00:42.203488027 (198h20m13.89933678s) [upload=174795]: Upload finished. err: <nil> Duration: 24.752536236s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.571880679s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:01:10 UTC) orbiter warning: context deadline exceeded Duration: 3.571903122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [...........XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX] duration: 30.007129854s trace: https://jaeger.dev.storj.io/trace/69cc3cc02350ba23 FAIL (2025-11-07 18:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: 5HEE5776LOPMUBWY23P3NMIOQ6GXZHDGLZR2WIEUPZDG6SCNJWHQ) context deadline exceeded; piecestore: (Node ID: 18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, Piece ID: UQ2DVP2OCJBXP237XF3PEIUG5TLRMJ7HHVL6DZJLO2VQDWW5KYIQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: HXWDF5J6QKCBRWMW47DKSEI7TE5BSISOHNRBQV3WQH74ZZRQWH2A) context deadline exceeded ======LOGS====== Nov 7 18:00:38.456604225 (198h20m10.152453018s) [upload=174795]: Starting upload Nov 7 18:00:38.456697002 (198h20m10.152545795s) [upload=174795]: Done starting upload Nov 7 18:00:38.456746345 (198h20m10.152595108s) [upload=174795]: Uploading segments... Nov 7 18:00:38.457038933 (198h20m10.152887686s) [upload=174795]: Got next segment. Inline: false Nov 7 18:00:38.457129084 (198h20m10.152977827s) [upload=174795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:00:38.457138502 (198h20m10.152987255s) [upload=174795]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:00:38.747435703 (198h20m10.443284456s) [seg_pos={0 0}, upload=174795]: Begin upload segment... Nov 7 18:00:38.747501588 (198h20m10.443350341s) [seg_pos={0 0}, upload=174795]: Done begin upload segment. Nov 7 18:00:38.747549399 (198h20m10.443398152s) [noise=true , address=145.79.227.22:4510 , piece_id=X3IWHVTRRLDREQYN2HH64URYRSBCZSKHP453DVTN62IDMTMPSTOQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.751832135 (198h20m10.447680888s) [noise=true , address=45.91.116.9:4263 , piece_id=HXWDF5J6QKCBRWMW47DKSEI7TE5BSISOHNRBQV3WQH74ZZRQWH2A, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.753088377 (198h20m10.44893714s) [noise=true , address=45.91.116.14:4190 , piece_id=JF3IIN4AMZSKMROWP5PDJR5UJDFCOLMEYC3NZBS4R4WVNYO6H4QA, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.753933676 (198h20m10.449782429s) [noise=true , address=45.91.116.8:4048 , piece_id=XWLMXVUY3RV2KJS7M2V2IAB7XNYWOCMSMURGGNU33ZEXIRM2ZYLA, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.754526735 (198h20m10.450375488s) [noise=true , address=145.79.227.30:4761 , piece_id=5DO2RD74OX4PT5LNN2IPQOV6LT6PBA3PYMC3IODXFWSETAUAKUOQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.755097332 (198h20m10.450946086s) [noise=true , address=145.79.227.32:4836 , piece_id=TTXQIIYFDHFS43UQH5LBZONHX7YJLXRS6DSAJ7SRQW3DD46A2P7A, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.755623805 (198h20m10.451472568s) [noise=true , address=145.79.227.28:4725 , piece_id=6XS2BOMMKE7TADHNPT5ZGTRUUGSAYSY7BBQMHIIUFDX6KI5Q3DWA, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.756248655 (198h20m10.452097408s) [noise=true , address=45.91.116.7:4010 , piece_id=K73CC7XUA2AHMXRADZEWBSVHY5IXXYXEXEJI4VWV76OQ2VTVYOEQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.756815013 (198h20m10.452663766s) [noise=true , address=77.90.31.21:4879 , piece_id=4CLWXXBVVSAVKUPM72Q5P2OLVPWIDVOEMM2GAQO6OXHJSURL7DTQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.758837093 (198h20m10.454685856s) [noise=true , address=145.79.227.25:4617 , piece_id=MWZNA4FOK4YHJRGPDKJVE6A5QVRQARIVJD2SAAX2B5MQLOPJBU5A, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.759687011 (198h20m10.455535764s) [noise=true , address=145.79.227.15:4876 , piece_id=GN5GEGC6FPXUTDLN6XFY5CPQMIGRPTDHIO7MLROBNVMGSKPZA3IQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.760284570 (198h20m10.456133313s) [noise=true , address=77.90.31.15:4698 , piece_id=5HEE5776LOPMUBWY23P3NMIOQ6GXZHDGLZR2WIEUPZDG6SCNJWHQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.760973431 (198h20m10.456822194s) [noise=true , address=145.79.227.13:4803 , piece_id=KW3A7RGL77V3I4IR7L36GFOHWIWQZNTPPSDUPIHQ7K4JMPGBFUMQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.761679346 (198h20m10.457528099s) [noise=true , address=145.79.227.10:4403 , piece_id=ELMFQMWILFX5NNE4ECLT6QPN3GE4VW2ZQMFEHVF4QBVEU3P6JBDQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.763257020 (198h20m10.459105773s) [noise=true , address=145.79.227.26:4655 , piece_id=5X6ABUUXWAPIKTOAVLTZRZRJYTHTYTG36SRJ5NEHVQ6NVX5FEDUQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:38.764773147 (198h20m10.46062191s) [noise=true , address=77.90.31.17:4772 , piece_id=UQ2DVP2OCJBXP237XF3PEIUG5TLRMJ7HHVL6DZJLO2VQDWW5KYIQ, seg_pos={0 0}, upload=174795]: Uploading piece... Nov 7 18:00:40.560729193 (198h20m12.256577936s) [upload=174795]: Next returned nil segment Nov 7 18:00:40.560743740 (198h20m12.256592493s) [upload=174795]: Waiting for error group managing segments... Nov 7 18:00:40.863633679 (198h20m12.559482442s) [noise=true , address=145.79.227.32:4836 , piece_id=TTXQIIYFDHFS43UQH5LBZONHX7YJLXRS6DSAJ7SRQW3DD46A2P7A, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:40.940082104 (198h20m12.635930857s) [noise=true , address=145.79.227.26:4655 , piece_id=5X6ABUUXWAPIKTOAVLTZRZRJYTHTYTG36SRJ5NEHVQ6NVX5FEDUQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:40.983226180 (198h20m12.679074933s) [noise=true , address=145.79.227.10:4403 , piece_id=ELMFQMWILFX5NNE4ECLT6QPN3GE4VW2ZQMFEHVF4QBVEU3P6JBDQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.034203803 (198h20m12.730052556s) [noise=true , address=145.79.227.25:4617 , piece_id=MWZNA4FOK4YHJRGPDKJVE6A5QVRQARIVJD2SAAX2B5MQLOPJBU5A, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.053008571 (198h20m12.748857324s) [noise=true , address=145.79.227.30:4761 , piece_id=5DO2RD74OX4PT5LNN2IPQOV6LT6PBA3PYMC3IODXFWSETAUAKUOQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.070400158 (198h20m12.766248921s) [noise=true , address=145.79.227.13:4803 , piece_id=KW3A7RGL77V3I4IR7L36GFOHWIWQZNTPPSDUPIHQ7K4JMPGBFUMQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.415706468 (198h20m13.111555221s) [noise=true , address=77.90.31.15:4698 , piece_id=5HEE5776LOPMUBWY23P3NMIOQ6GXZHDGLZR2WIEUPZDG6SCNJWHQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.417404581 (198h20m13.113253334s) [noise=true , address=145.79.227.15:4876 , piece_id=GN5GEGC6FPXUTDLN6XFY5CPQMIGRPTDHIO7MLROBNVMGSKPZA3IQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.469060125 (198h20m13.164908878s) [noise=true , address=145.79.227.28:4725 , piece_id=6XS2BOMMKE7TADHNPT5ZGTRUUGSAYSY7BBQMHIIUFDX6KI5Q3DWA, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.498378913 (198h20m13.194227666s) [noise=true , address=77.90.31.17:4772 , piece_id=UQ2DVP2OCJBXP237XF3PEIUG5TLRMJ7HHVL6DZJLO2VQDWW5KYIQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.731346175 (198h20m13.427194928s) [noise=true , address=145.79.227.22:4510 , piece_id=X3IWHVTRRLDREQYN2HH64URYRSBCZSKHP453DVTN62IDMTMPSTOQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.845993869 (198h20m13.541842622s) [noise=true , address=45.91.116.9:4263 , piece_id=HXWDF5J6QKCBRWMW47DKSEI7TE5BSISOHNRBQV3WQH74ZZRQWH2A, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.872505804 (198h20m13.568354567s) [noise=true , address=45.91.116.14:4190 , piece_id=JF3IIN4AMZSKMROWP5PDJR5UJDFCOLMEYC3NZBS4R4WVNYO6H4QA, seg_pos={0 0}, upload=174795]: Done uploading piece. err: <nil> Nov 7 18:00:41.872553955 (198h20m13.568402708s) [seg_pos={0 0}, upload=174795]: Segment reached optimal threshold of 13 pieces. Nov 7 18:00:41.872825513 (198h20m13.568674276s) [noise=true , address=77.90.31.21:4879 , piece_id=4CLWXXBVVSAVKUPM72Q5P2OLVPWIDVOEMM2GAQO6OXHJSURL7DTQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 188.166.225.170:40814->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:40814->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40814->77.90.31.21:4879: use of closed network connection Nov 7 18:00:41.872835481 (198h20m13.568684234s) [noise=true , address=77.90.31.21:4879 , piece_id=4CLWXXBVVSAVKUPM72Q5P2OLVPWIDVOEMM2GAQO6OXHJSURL7DTQ, seg_pos={0 0}, upload=174795]: Long tail cancellation due to optimal threshold Nov 7 18:00:41.872913841 (198h20m13.568762604s) [noise=true , address=45.91.116.8:4048 , piece_id=XWLMXVUY3RV2KJS7M2V2IAB7XNYWOCMSMURGGNU33ZEXIRM2ZYLA, seg_pos={0 0}, upload=174795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:41.872947094 (198h20m13.568795848s) [noise=true , address=45.91.116.8:4048 , piece_id=XWLMXVUY3RV2KJS7M2V2IAB7XNYWOCMSMURGGNU33ZEXIRM2ZYLA, seg_pos={0 0}, upload=174795]: Long tail cancellation due to optimal threshold Nov 7 18:00:41.873095577 (198h20m13.56894434s) [noise=true , address=45.91.116.7:4010 , piece_id=K73CC7XUA2AHMXRADZEWBSVHY5IXXYXEXEJI4VWV76OQ2VTVYOEQ, seg_pos={0 0}, upload=174795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:41.873107099 (198h20m13.568955852s) [noise=true , address=45.91.116.7:4010 , piece_id=K73CC7XUA2AHMXRADZEWBSVHY5IXXYXEXEJI4VWV76OQ2VTVYOEQ, seg_pos={0 0}, upload=174795]: Long tail cancellation due to optimal threshold Nov 7 18:00:41.873132047 (198h20m13.5689808s) [seg_pos={0 0}, upload=174795]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:00:41.873226467 (198h20m13.56907522s) [upload=174795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:00:41.873234792 (198h20m13.569083545s) [upload=174795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:00:42.203462148 (198h20m13.899310911s) [upload=174795]: Done uploading segments... Nov 7 18:00:42.203488027 (198h20m13.89933678s) [upload=174795]: Upload finished. err: <nil> Duration: 30.007148591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [............XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXX] duration: 30.01098767s trace: https://jaeger.dev.storj.io/trace/562a314ab8e6ba7 FAIL (2025-11-07 18:00:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: DFMUTP456YEMEZ3YTBEHCKCDC5YT5AEIOQMUTCUHSXSMQFQPMFNQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: AZUDZXGCTOP2KXJHOMBO236PAPSTZIGLYNMFHW3I2Q4DQ3KIA3BA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4BNQZ3JVFXLHJFX27HVSRNWCP422JUQWCNE3WTIVWZDASLWS4U5Q) context deadline exceeded ======LOGS====== Nov 7 18:00:08.146249143 (198h19m39.842097896s) [upload=174786]: Starting upload Nov 7 18:00:08.146340848 (198h19m39.842189601s) [upload=174786]: Done starting upload Nov 7 18:00:08.146637414 (198h19m39.842486167s) [upload=174786]: Uploading segments... Nov 7 18:00:08.146707357 (198h19m39.84255611s) [upload=174786]: Got next segment. Inline: false Nov 7 18:00:08.146767881 (198h19m39.842616634s) [upload=174786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:00:08.146774584 (198h19m39.842623337s) [upload=174786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:00:08.394163326 (198h19m40.090012089s) [seg_pos={0 0}, upload=174786]: Begin upload segment... Nov 7 18:00:08.394228781 (198h19m40.090077544s) [seg_pos={0 0}, upload=174786]: Done begin upload segment. Nov 7 18:00:08.394270921 (198h19m40.090119684s) [noise=true , address=45.91.116.13:4153 , piece_id=DFMUTP456YEMEZ3YTBEHCKCDC5YT5AEIOQMUTCUHSXSMQFQPMFNQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.395537633 (198h19m40.091386386s) [noise=true , address=145.79.227.25:4618 , piece_id=3D7YDZNWT4P3JCXPSXIAOO73UNP4PIZETZFLTNQSR6MFW55ONVDQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.396059417 (198h19m40.09190818s) [noise=true , address=77.90.31.13:4626 , piece_id=4T6Q4XNVK2JBI47TOIVFQSUL5IBQNFP26H3ULZLE5I257GKE2DFQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.396585990 (198h19m40.092434753s) [noise=true , address=145.79.227.10:4403 , piece_id=Q4OCCYHTUAPGC6SDIIWZ357MMEWJGAGR6NZJTAEG65KENDJ6GGQA, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.397048741 (198h19m40.092897494s) [noise=true , address=45.91.116.14:4191 , piece_id=AZUDZXGCTOP2KXJHOMBO236PAPSTZIGLYNMFHW3I2Q4DQ3KIA3BA, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.397544365 (198h19m40.093393128s) [noise=true , address=145.79.227.4:4186 , piece_id=5IKVJQRQYSAKKOVANUU7QSJ5MWOVXT6KCFQVU63VT4BSY7AKFQIQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.398001716 (198h19m40.093850469s) [noise=true , address=45.91.116.12:4226 , piece_id=4BNQZ3JVFXLHJFX27HVSRNWCP422JUQWCNE3WTIVWZDASLWS4U5Q, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.398490897 (198h19m40.09433965s) [noise=true , address=145.79.227.11:4730 , piece_id=2WSV22YQ2E3RVNX3RXRITQX5RTEHCLZW3ELL2DF7BU3RZQTHSMEQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.398938449 (198h19m40.094787233s) [noise=true , address=145.79.227.15:4875 , piece_id=HR46XLRKSP2UQOY5HG47QRAHPSDQGDAOR7UH554NTSSSPSQ2NRRA, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.399562568 (198h19m40.095411321s) [noise=true , address=145.79.227.24:4581 , piece_id=KZFQIE6TNXNAXITKT2T5JHRNOZKCIOAZOVAQJN36Q26UN4DJPHUQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.399995382 (198h19m40.095844135s) [noise=true , address=77.90.31.10:4518 , piece_id=RP4B6QQBXK6RNDSCETBRWPK67MO7HUZPAOGNFHPJHD7ZT4RBPADA, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.400388621 (198h19m40.096237374s) [noise=true , address=45.91.116.5:4299 , piece_id=MOS2QLWRFRRGRJAQHYL5462BVKO53J4IUMWMU3MSHZHZXG3Z342Q, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.400819010 (198h19m40.096667763s) [noise=true , address=45.91.116.7:4011 , piece_id=FRWGB55BCAHT7MCR6LY4I7RZHUHDEYXDAKE6OQWBBACYUHYN44KA, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.401243609 (198h19m40.097092362s) [noise=true , address=45.91.116.11:4117 , piece_id=FQMID33I2HPOPHQTOYYBYV6DBTGYXJ7PDDMH35J7CIMXQVCSQW4Q, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.401712702 (198h19m40.097561466s) [noise=true , address=145.79.227.30:4762 , piece_id=TUWFNJT23VJL23SWFZIRBEKBT4PFE3HBNPN35PHAHN5LQMIL7ZVQ, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:08.402212113 (198h19m40.098060866s) [noise=true , address=145.79.227.9:4365 , piece_id=WXGVXJBQHPT5X4HOGDIWIB7PERGR33YJITZAGWLPU3CWIWHHBD3Q, seg_pos={0 0}, upload=174786]: Uploading piece... Nov 7 18:00:10.165441731 (198h19m41.861290484s) [upload=174786]: Next returned nil segment Nov 7 18:00:10.165461058 (198h19m41.861309811s) [upload=174786]: Waiting for error group managing segments... Nov 7 18:00:10.585367252 (198h19m42.281216005s) [noise=true , address=145.79.227.15:4875 , piece_id=HR46XLRKSP2UQOY5HG47QRAHPSDQGDAOR7UH554NTSSSPSQ2NRRA, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.594563633 (198h19m42.290412387s) [noise=true , address=145.79.227.9:4365 , piece_id=WXGVXJBQHPT5X4HOGDIWIB7PERGR33YJITZAGWLPU3CWIWHHBD3Q, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.596078979 (198h19m42.291927732s) [noise=true , address=145.79.227.11:4730 , piece_id=2WSV22YQ2E3RVNX3RXRITQX5RTEHCLZW3ELL2DF7BU3RZQTHSMEQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.596722775 (198h19m42.292571528s) [noise=true , address=145.79.227.25:4618 , piece_id=3D7YDZNWT4P3JCXPSXIAOO73UNP4PIZETZFLTNQSR6MFW55ONVDQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.604853738 (198h19m42.300702491s) [noise=true , address=145.79.227.24:4581 , piece_id=KZFQIE6TNXNAXITKT2T5JHRNOZKCIOAZOVAQJN36Q26UN4DJPHUQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.624879341 (198h19m42.320728094s) [noise=true , address=145.79.227.4:4186 , piece_id=5IKVJQRQYSAKKOVANUU7QSJ5MWOVXT6KCFQVU63VT4BSY7AKFQIQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.689406435 (198h19m42.385255188s) [noise=true , address=145.79.227.30:4762 , piece_id=TUWFNJT23VJL23SWFZIRBEKBT4PFE3HBNPN35PHAHN5LQMIL7ZVQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.694611787 (198h19m42.39046054s) [noise=true , address=45.91.116.12:4226 , piece_id=4BNQZ3JVFXLHJFX27HVSRNWCP422JUQWCNE3WTIVWZDASLWS4U5Q, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.764249964 (198h19m42.460098727s) [noise=true , address=45.91.116.14:4191 , piece_id=AZUDZXGCTOP2KXJHOMBO236PAPSTZIGLYNMFHW3I2Q4DQ3KIA3BA, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.791775909 (198h19m42.487624672s) [noise=true , address=45.91.116.5:4299 , piece_id=MOS2QLWRFRRGRJAQHYL5462BVKO53J4IUMWMU3MSHZHZXG3Z342Q, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.792469490 (198h19m42.488318243s) [noise=true , address=45.91.116.13:4153 , piece_id=DFMUTP456YEMEZ3YTBEHCKCDC5YT5AEIOQMUTCUHSXSMQFQPMFNQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.800914361 (198h19m42.496763124s) [noise=true , address=45.91.116.11:4117 , piece_id=FQMID33I2HPOPHQTOYYBYV6DBTGYXJ7PDDMH35J7CIMXQVCSQW4Q, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.865019852 (198h19m42.560868605s) [noise=true , address=77.90.31.10:4518 , piece_id=RP4B6QQBXK6RNDSCETBRWPK67MO7HUZPAOGNFHPJHD7ZT4RBPADA, seg_pos={0 0}, upload=174786]: Done uploading piece. err: <nil> Nov 7 18:00:10.865049359 (198h19m42.560898122s) [seg_pos={0 0}, upload=174786]: Segment reached optimal threshold of 13 pieces. Nov 7 18:00:10.865315264 (198h19m42.561164018s) [noise=true , address=77.90.31.13:4626 , piece_id=4T6Q4XNVK2JBI47TOIVFQSUL5IBQNFP26H3ULZLE5I257GKE2DFQ, seg_pos={0 0}, upload=174786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:10.865325694 (198h19m42.561174457s) [noise=true , address=77.90.31.13:4626 , piece_id=4T6Q4XNVK2JBI47TOIVFQSUL5IBQNFP26H3ULZLE5I257GKE2DFQ, seg_pos={0 0}, upload=174786]: Long tail cancellation due to optimal threshold Nov 7 18:00:10.866750928 (198h19m42.562599691s) [noise=true , address=145.79.227.10:4403 , piece_id=Q4OCCYHTUAPGC6SDIIWZ357MMEWJGAGR6NZJTAEG65KENDJ6GGQA, seg_pos={0 0}, upload=174786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:10.866762821 (198h19m42.562611574s) [noise=true , address=145.79.227.10:4403 , piece_id=Q4OCCYHTUAPGC6SDIIWZ357MMEWJGAGR6NZJTAEG65KENDJ6GGQA, seg_pos={0 0}, upload=174786]: Long tail cancellation due to optimal threshold Nov 7 18:00:10.867551512 (198h19m42.563400265s) [noise=true , address=45.91.116.7:4011 , piece_id=FRWGB55BCAHT7MCR6LY4I7RZHUHDEYXDAKE6OQWBBACYUHYN44KA, seg_pos={0 0}, upload=174786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Nov 7 18:00:10.867578404 (198h19m42.563427167s) [noise=true , address=45.91.116.7:4011 , piece_id=FRWGB55BCAHT7MCR6LY4I7RZHUHDEYXDAKE6OQWBBACYUHYN44KA, seg_pos={0 0}, upload=174786]: Long tail cancellation due to optimal threshold Nov 7 18:00:10.867623109 (198h19m42.563471862s) [seg_pos={0 0}, upload=174786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:00:10.867778205 (198h19m42.563626958s) [upload=174786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:00:10.867788514 (198h19m42.563637267s) [upload=174786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:00:11.208809303 (198h19m42.904658056s) [upload=174786]: Done uploading segments... Nov 7 18:00:11.208836605 (198h19m42.904685358s) [upload=174786]: Upload finished. err: <nil> Duration: 30.011029249s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.76944834s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:00:15 UTC) orbiter warning: context deadline exceeded Duration: 3.769478489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [.............XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XX] duration: 30.01257992s trace: https://jaeger.dev.storj.io/trace/4547efbba0cec63 FAIL (2025-11-07 18:00:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h19m5.433519417s) [upload=174780]: Uploading segments... Nov 7 17:59:33.737978200 (198h19m5.433826953s) [upload=174780]: Got next segment. Inline: false Nov 7 17:59:33.738029667 (198h19m5.43387841s) [upload=174780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:59:33.738038845 (198h19m5.433887598s) [upload=174780]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:59:33.987851382 (198h19m5.683700136s) [seg_pos={0 0}, upload=174780]: Begin upload segment... Nov 7 17:59:33.987912569 (198h19m5.683761322s) [seg_pos={0 0}, upload=174780]: Done begin upload segment. Nov 7 17:59:33.987951393 (198h19m5.683800146s) [noise=true , address=45.91.116.4:4938 , piece_id=JBN22J73YWKPIJWSPVCF6HUYQJVDP5O37W6MAE3LCUMDHXMOCZUA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.988018170 (198h19m5.683866923s) [noise=true , address=145.79.227.8:4330 , piece_id=WZN7W22APNFBFLBG4N4THC7TGY7CPEU5DKA5CZNWBFWHEE7HHUNA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.988635446 (198h19m5.684484199s) [noise=true , address=145.79.227.14:4837 , piece_id=GZK3KVFDYKUIC3ZH3BHRG5KRYAIJUTEIRMW2WVILBFB2OVUQBIHA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.989327544 (198h19m5.685176297s) [noise=true , address=145.79.227.20:4438 , piece_id=WP63H7OS5W66RSJRPSIWADCVBWU5JPLCSJ5H6MCHRUQFGKBD3ALQ, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.989908850 (198h19m5.685757603s) [noise=true , address=77.90.31.14:4661 , piece_id=5FHR7LQHB6BMSG6ZES3HM7KIIV2EMLQZJSESB6FRF56GJHNVW4WA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.990486771 (198h19m5.686335524s) [noise=true , address=77.90.31.11:4555 , piece_id=L2EQMJNBV4F2E3NJWCURMCBW2AX3PDSLUGWHTZWOEZPJX6KXTYDQ, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.991257418 (198h19m5.687106171s) [noise=true , address=77.90.31.16:4735 , piece_id=ZD4VB74E54TSPA6TK5OHPMKA3V4LFG47TQRMV2AQSEZOSNYL3FZA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.992584295 (198h19m5.688433048s) [noise=true , address=77.90.31.17:4770 , piece_id=ALLKX3GLE6A4F2PXKUA5DOUNPGH2LSOS43GID4RTUJ27XOFD2XZQ, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.992720995 (198h19m5.688569748s) [noise=true , address=145.79.227.27:4690 , piece_id=TENSEN5STUMC227FQADP6RWK4WQYDI526VV23GRFTBAQMG6YP2DA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.994864637 (198h19m5.690713381s) [noise=true , address=145.79.227.11:4730 , piece_id=XBMKYGVG5HG77QNDY4RVSEUDSZPSXMROOUBGGVOXAFITIUQS42XQ, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.995713884 (198h19m5.691562657s) [noise=true , address=145.79.227.25:4618 , piece_id=4PFMW7NOHYWEDOZVGWLPWPDH5RZF3ZCBKNQS6KNGMM7IFFF5HDHQ, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.996320800 (198h19m5.692169553s) [noise=true , address=145.79.227.9:4366 , piece_id=7AFQVW5XTEIB4VL57IPYLJBZ35WEIQBU2OFNIGDOBICCST36JDAA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:33.997385096 (198h19m5.693233839s) [noise=true , address=145.79.227.16:4911 , piece_id=NGBJEXTLDX5UE26CS6BVOLHJSPM37IFJZJHT7A4CMLU3LKUOZNVQ, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:34.000394227 (198h19m5.69624301s) [noise=true , address=77.90.31.21:4877 , piece_id=J53BVKA5WJMMONYX5JI75BHYW6QO3W7BL4D2T52UNADZLX46R6GA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:34.003462720 (198h19m5.699311493s) [noise=true , address=45.91.116.12:4227 , piece_id=ZQMZOIEWREO2CGEDAWJ66CXOEPVTPLA5THMMZEHZOVS4IJQAISCA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:34.005706883 (198h19m5.701555657s) [noise=true , address=45.91.116.11:4117 , piece_id=XB56N7NIMWIO2GTMOXNJWKQXRDHK7BUTWHSOBMNTNIJG547WVWFA, seg_pos={0 0}, upload=174780]: Uploading piece... Nov 7 17:59:35.817070748 (198h19m7.512919501s) [upload=174780]: Next returned nil segment Nov 7 17:59:35.817090176 (198h19m7.512938929s) [upload=174780]: Waiting for error group managing segments... Nov 7 17:59:36.216897196 (198h19m7.912745949s) [noise=true , address=45.91.116.4:4938 , piece_id=JBN22J73YWKPIJWSPVCF6HUYQJVDP5O37W6MAE3LCUMDHXMOCZUA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.242106710 (198h19m7.937955463s) [noise=true , address=77.90.31.11:4555 , piece_id=L2EQMJNBV4F2E3NJWCURMCBW2AX3PDSLUGWHTZWOEZPJX6KXTYDQ, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.257222784 (198h19m7.953071528s) [noise=true , address=145.79.227.25:4618 , piece_id=4PFMW7NOHYWEDOZVGWLPWPDH5RZF3ZCBKNQS6KNGMM7IFFF5HDHQ, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.263822943 (198h19m7.959671696s) [noise=true , address=77.90.31.14:4661 , piece_id=5FHR7LQHB6BMSG6ZES3HM7KIIV2EMLQZJSESB6FRF56GJHNVW4WA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.272982184 (198h19m7.968830927s) [noise=true , address=77.90.31.16:4735 , piece_id=ZD4VB74E54TSPA6TK5OHPMKA3V4LFG47TQRMV2AQSEZOSNYL3FZA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.282704147 (198h19m7.97855289s) [noise=true , address=145.79.227.16:4911 , piece_id=NGBJEXTLDX5UE26CS6BVOLHJSPM37IFJZJHT7A4CMLU3LKUOZNVQ, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.312698122 (198h19m8.008546876s) [noise=true , address=145.79.227.27:4690 , piece_id=TENSEN5STUMC227FQADP6RWK4WQYDI526VV23GRFTBAQMG6YP2DA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.333974988 (198h19m8.029823731s) [noise=true , address=145.79.227.8:4330 , piece_id=WZN7W22APNFBFLBG4N4THC7TGY7CPEU5DKA5CZNWBFWHEE7HHUNA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.337260765 (198h19m8.033109518s) [noise=true , address=45.91.116.12:4227 , piece_id=ZQMZOIEWREO2CGEDAWJ66CXOEPVTPLA5THMMZEHZOVS4IJQAISCA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.351984422 (198h19m8.047833175s) [noise=true , address=145.79.227.14:4837 , piece_id=GZK3KVFDYKUIC3ZH3BHRG5KRYAIJUTEIRMW2WVILBFB2OVUQBIHA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.353271703 (198h19m8.049120456s) [noise=true , address=145.79.227.20:4438 , piece_id=WP63H7OS5W66RSJRPSIWADCVBWU5JPLCSJ5H6MCHRUQFGKBD3ALQ, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.367697663 (198h19m8.063546406s) [noise=true , address=145.79.227.11:4730 , piece_id=XBMKYGVG5HG77QNDY4RVSEUDSZPSXMROOUBGGVOXAFITIUQS42XQ, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.499092127 (198h19m8.19494088s) [noise=true , address=145.79.227.9:4366 , piece_id=7AFQVW5XTEIB4VL57IPYLJBZ35WEIQBU2OFNIGDOBICCST36JDAA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: <nil> Nov 7 17:59:36.499120161 (198h19m8.194968904s) [seg_pos={0 0}, upload=174780]: Segment reached optimal threshold of 13 pieces. Nov 7 17:59:36.499301225 (198h19m8.195149979s) [noise=true , address=45.91.116.11:4117 , piece_id=XB56N7NIMWIO2GTMOXNJWKQXRDHK7BUTWHSOBMNTNIJG547WVWFA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:35176->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:35176->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35176->45.91.116.11:4117: use of closed network connection Nov 7 17:59:36.499307748 (198h19m8.195156491s) [noise=true , address=45.91.116.11:4117 , piece_id=XB56N7NIMWIO2GTMOXNJWKQXRDHK7BUTWHSOBMNTNIJG547WVWFA, seg_pos={0 0}, upload=174780]: Long tail cancellation due to optimal threshold Nov 7 17:59:36.499424501 (198h19m8.195273254s) [noise=true , address=77.90.31.17:4770 , piece_id=ALLKX3GLE6A4F2PXKUA5DOUNPGH2LSOS43GID4RTUJ27XOFD2XZQ, seg_pos={0 0}, upload=174780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42062->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42062->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42062->77.90.31.17:4770: use of closed network connection Nov 7 17:59:36.499436072 (198h19m8.195284825s) [noise=true , address=77.90.31.17:4770 , piece_id=ALLKX3GLE6A4F2PXKUA5DOUNPGH2LSOS43GID4RTUJ27XOFD2XZQ, seg_pos={0 0}, upload=174780]: Long tail cancellation due to optimal threshold Nov 7 17:59:36.499535913 (198h19m8.195384706s) [noise=true , address=77.90.31.21:4877 , piece_id=J53BVKA5WJMMONYX5JI75BHYW6QO3W7BL4D2T52UNADZLX46R6GA, seg_pos={0 0}, upload=174780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Nov 7 17:59:36.499546923 (198h19m8.195395676s) [noise=true , address=77.90.31.21:4877 , piece_id=J53BVKA5WJMMONYX5JI75BHYW6QO3W7BL4D2T52UNADZLX46R6GA, seg_pos={0 0}, upload=174780]: Long tail cancellation due to optimal threshold Nov 7 17:59:36.499572653 (198h19m8.195421406s) [seg_pos={0 0}, upload=174780]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:59:36.499661713 (198h19m8.195510456s) [upload=174780]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:59:36.499668906 (198h19m8.195517649s) [upload=174780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:59:36.805492602 (198h19m8.501341365s) [upload=174780]: Done uploading segments... Nov 7 17:59:36.805517218 (198h19m8.501365961s) [upload=174780]: Upload finished. err: <nil> Duration: 30.012608215s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.441463195s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:59:47 UTC) orbiter warning: context deadline exceeded Duration: 3.441490386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [..............XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.X] duration: 30.002092156s trace: https://jaeger.dev.storj.io/trace/5b194f53deba1c56 FAIL (2025-11-07 17:59:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6FW5W5T5FSRIJ52OMZCWFBX6HE2B3ZDGIRSQPK25T3UCCPZHSAKA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JGV5GTKFY5WVZKC3VPQM3X4XTQ6R2Z3LRGUVBGE46IUK5YRFOZAA) context deadline exceeded ======LOGS====== v 7 17:58:57.731555365 (198h18m29.427404108s) [upload=174772]: Starting upload Nov 7 17:58:57.731614318 (198h18m29.427463071s) [upload=174772]: Done starting upload Nov 7 17:58:57.731797807 (198h18m29.42764655s) [upload=174772]: Uploading segments... Nov 7 17:58:57.731906283 (198h18m29.427755036s) [upload=174772]: Got next segment. Inline: false Nov 7 17:58:57.731990885 (198h18m29.427839668s) [upload=174772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:58:57.732020922 (198h18m29.427869675s) [upload=174772]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:58:57.980334054 (198h18m29.676182837s) [seg_pos={0 0}, upload=174772]: Begin upload segment... Nov 7 17:58:57.980404939 (198h18m29.676253692s) [seg_pos={0 0}, upload=174772]: Done begin upload segment. Nov 7 17:58:57.980443542 (198h18m29.676292296s) [noise=true , address=45.91.116.8:4048 , piece_id=6FW5W5T5FSRIJ52OMZCWFBX6HE2B3ZDGIRSQPK25T3UCCPZHSAKA, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.980516983 (198h18m29.676365726s) [noise=true , address=45.91.116.14:4191 , piece_id=JGV5GTKFY5WVZKC3VPQM3X4XTQ6R2Z3LRGUVBGE46IUK5YRFOZAA, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.981086377 (198h18m29.67693513s) [noise=true , address=77.90.31.15:4697 , piece_id=YZJXMU4ZSHMV65QARZUJJRWMFST7OJW72M3XW426X7XHVPGY3BKQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.981441663 (198h18m29.677290406s) [noise=true , address=145.79.227.31:4798 , piece_id=J4UB6QDFCQDXX52EDFXLLCSP6F6XCBVXJCGRI6H4IRHFP4XKKLVQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.981448496 (198h18m29.677297249s) [noise=true , address=145.79.227.13:4802 , piece_id=ZO5XDWEWGYQPMAVD2LUAIOKPIME2T6S7FPEIG3SAGCD6MNMHEJUQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.981730574 (198h18m29.677579317s) [noise=true , address=145.79.227.11:4732 , piece_id=WWIG2RJV4JJVLLNCEISW3WISXUZXC64Q7AJKC2KBESRXMRMGXEEQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.981964309 (198h18m29.677813062s) [noise=true , address=145.79.227.18:4005 , piece_id=QXJS4KLGU3JYC2SHDBICINHAILG4H234TDZLQI4OXMKJOUWIOZKA, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.982125496 (198h18m29.677974249s) [noise=true , address=77.90.31.18:4807 , piece_id=PGV4B2VYQW7NHZTMAOIB7K355D6XZUUY7XIVMMPCMHPIE5WYUMYQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.982297023 (198h18m29.678145776s) [noise=true , address=145.79.227.17:4972 , piece_id=YWTOG5FOROU3WP75ENQTEBUZYL33H3BKHQTQD7ESODLZ2AN22XGQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.982554573 (198h18m29.678403326s) [noise=true , address=145.79.227.27:4691 , piece_id=HOL43WL5R4RR7CMDU4UIONFEM5NDYMX6FQKJJ2S7V2II2Z5W72BQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.982642030 (198h18m29.678490783s) [noise=true , address=145.79.227.22:4511 , piece_id=HUVFICKV5K3UUKL6UOXBPAIBBYNK66DNQEDYBIMFZI6DCGZL5GCQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.983040929 (198h18m29.678889682s) [noise=true , address=145.79.227.14:4840 , piece_id=7WYKQH7CQL75RUZV25T65TGLNOQ5KEJX725RYU6W55CLKNXRS33Q, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.983164074 (198h18m29.679012827s) [noise=true , address=145.79.227.20:4438 , piece_id=FZ5YDLCHUJNUS5O5TS2JSOEM2Q32RXHA4NI3WAACHFCH6ZWQO56A, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.983426954 (198h18m29.679275707s) [noise=true , address=45.91.116.13:4153 , piece_id=IPLK5YLXDU7K75PL7FQBHTYP4BQAZPUPWRAGVB5NVRQBAHDFOVNQ, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.983506566 (198h18m29.679355319s) [noise=true , address=45.91.116.4:4939 , piece_id=X35JCLL77PPHUPBX6V63QLR5RYHAQTAOYLAPKDRUOVSZXZNMEA5A, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:57.983876480 (198h18m29.679725233s) [noise=true , address=45.91.116.7:4010 , piece_id=OBCKKL7ZKRZMKLNTTZBU7LLDORLOG4G5S27ST7WK55HQG23RLMYA, seg_pos={0 0}, upload=174772]: Uploading piece... Nov 7 17:58:59.643937612 (198h18m31.339786365s) [upload=174772]: Next returned nil segment Nov 7 17:58:59.643953903 (198h18m31.339802646s) [upload=174772]: Waiting for error group managing segments... Nov 7 17:58:59.933223647 (198h18m31.6290724s) [noise=true , address=145.79.227.20:4438 , piece_id=FZ5YDLCHUJNUS5O5TS2JSOEM2Q32RXHA4NI3WAACHFCH6ZWQO56A, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:58:59.991932822 (198h18m31.687781575s) [noise=true , address=145.79.227.17:4972 , piece_id=YWTOG5FOROU3WP75ENQTEBUZYL33H3BKHQTQD7ESODLZ2AN22XGQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.017537569 (198h18m31.713386322s) [noise=true , address=145.79.227.11:4732 , piece_id=WWIG2RJV4JJVLLNCEISW3WISXUZXC64Q7AJKC2KBESRXMRMGXEEQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.024376100 (198h18m31.720224863s) [noise=true , address=145.79.227.18:4005 , piece_id=QXJS4KLGU3JYC2SHDBICINHAILG4H234TDZLQI4OXMKJOUWIOZKA, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.026455009 (198h18m31.722303762s) [noise=true , address=145.79.227.31:4798 , piece_id=J4UB6QDFCQDXX52EDFXLLCSP6F6XCBVXJCGRI6H4IRHFP4XKKLVQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.099793952 (198h18m31.795642705s) [noise=true , address=145.79.227.13:4802 , piece_id=ZO5XDWEWGYQPMAVD2LUAIOKPIME2T6S7FPEIG3SAGCD6MNMHEJUQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.124850365 (198h18m31.820699118s) [noise=true , address=145.79.227.22:4511 , piece_id=HUVFICKV5K3UUKL6UOXBPAIBBYNK66DNQEDYBIMFZI6DCGZL5GCQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.170329788 (198h18m31.866178541s) [noise=true , address=145.79.227.27:4691 , piece_id=HOL43WL5R4RR7CMDU4UIONFEM5NDYMX6FQKJJ2S7V2II2Z5W72BQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.192703974 (198h18m31.888552727s) [noise=true , address=145.79.227.14:4840 , piece_id=7WYKQH7CQL75RUZV25T65TGLNOQ5KEJX725RYU6W55CLKNXRS33Q, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.265417105 (198h18m31.961265869s) [noise=true , address=77.90.31.15:4697 , piece_id=YZJXMU4ZSHMV65QARZUJJRWMFST7OJW72M3XW426X7XHVPGY3BKQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.270233838 (198h18m31.966082581s) [noise=true , address=45.91.116.8:4048 , piece_id=6FW5W5T5FSRIJ52OMZCWFBX6HE2B3ZDGIRSQPK25T3UCCPZHSAKA, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.283292474 (198h18m31.979141227s) [noise=true , address=45.91.116.14:4191 , piece_id=JGV5GTKFY5WVZKC3VPQM3X4XTQ6R2Z3LRGUVBGE46IUK5YRFOZAA, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.285839855 (198h18m31.981688618s) [noise=true , address=45.91.116.7:4010 , piece_id=OBCKKL7ZKRZMKLNTTZBU7LLDORLOG4G5S27ST7WK55HQG23RLMYA, seg_pos={0 0}, upload=174772]: Done uploading piece. err: <nil> Nov 7 17:59:00.285860264 (198h18m31.981709017s) [seg_pos={0 0}, upload=174772]: Segment reached optimal threshold of 13 pieces. Nov 7 17:59:00.286114017 (198h18m31.98196277s) [noise=true , address=45.91.116.13:4153 , piece_id=IPLK5YLXDU7K75PL7FQBHTYP4BQAZPUPWRAGVB5NVRQBAHDFOVNQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 17:59:00.286122063 (198h18m31.981970816s) [noise=true , address=45.91.116.13:4153 , piece_id=IPLK5YLXDU7K75PL7FQBHTYP4BQAZPUPWRAGVB5NVRQBAHDFOVNQ, seg_pos={0 0}, upload=174772]: Long tail cancellation due to optimal threshold Nov 7 17:59:00.286420230 (198h18m31.982268983s) [noise=true , address=77.90.31.18:4807 , piece_id=PGV4B2VYQW7NHZTMAOIB7K355D6XZUUY7XIVMMPCMHPIE5WYUMYQ, seg_pos={0 0}, upload=174772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Nov 7 17:59:00.286428867 (198h18m31.98227762s) [noise=true , address=77.90.31.18:4807 , piece_id=PGV4B2VYQW7NHZTMAOIB7K355D6XZUUY7XIVMMPCMHPIE5WYUMYQ, seg_pos={0 0}, upload=174772]: Long tail cancellation due to optimal threshold Nov 7 17:59:00.286432804 (198h18m31.982281567s) [noise=true , address=45.91.116.4:4939 , piece_id=X35JCLL77PPHUPBX6V63QLR5RYHAQTAOYLAPKDRUOVSZXZNMEA5A, seg_pos={0 0}, upload=174772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44602->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44602->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44602->45.91.116.4:4939: use of closed network connection Nov 7 17:59:00.286478020 (198h18m31.982326773s) [noise=true , address=45.91.116.4:4939 , piece_id=X35JCLL77PPHUPBX6V63QLR5RYHAQTAOYLAPKDRUOVSZXZNMEA5A, seg_pos={0 0}, upload=174772]: Long tail cancellation due to optimal threshold Nov 7 17:59:00.286517405 (198h18m31.982366168s) [seg_pos={0 0}, upload=174772]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:59:00.286678001 (198h18m31.982526764s) [upload=174772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:59:00.286686858 (198h18m31.982535611s) [upload=174772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:59:00.591561295 (198h18m32.287410048s) [upload=174772]: Done uploading segments... Nov 7 17:59:00.591616210 (198h18m32.287464963s) [upload=174772]: Upload finished. err: <nil> Duration: 30.002117844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.587512615s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:59:19 UTC) orbiter warning: context deadline exceeded Duration: 3.587540698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.758754078s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:58:53 UTC) orbiter warning: context deadline exceeded Duration: 3.758825053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.492623824s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:58:36 UTC) orbiter warning: context deadline exceeded Duration: 3.492639794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [...............XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.] duration: 7.679757598s trace: https://jaeger.dev.storj.io/trace/2520db66a1bfb6d6 FAIL (2025-11-07 17:58:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 7 17:57:14.739375879 (198h16m46.435224632s) [upload=174748]: Starting upload Nov 7 17:57:14.739462223 (198h16m46.435310976s) [upload=174748]: Done starting upload Nov 7 17:57:14.739687112 (198h16m46.435535865s) [upload=174748]: Uploading segments... Nov 7 17:57:14.739763296 (198h16m46.43561205s) [upload=174748]: Got next segment. Inline: false Nov 7 17:57:14.739806308 (198h16m46.435655061s) [upload=174748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:57:14.739813603 (198h16m46.435662366s) [upload=174748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:57:14.986182943 (198h16m46.682031696s) [seg_pos={0 0}, upload=174748]: Begin upload segment... Nov 7 17:57:14.986240203 (198h16m46.682088966s) [seg_pos={0 0}, upload=174748]: Done begin upload segment. Nov 7 17:57:14.986343770 (198h16m46.682192553s) [noise=true , address=77.90.31.21:4879 , piece_id=PU77CJBPJOFVVWFA52ZM5GZIZASKUWDYS6P3L44IZNJ2LHDPYBSA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.986975212 (198h16m46.682823965s) [noise=true , address=77.90.31.12:4590 , piece_id=POZAAVNGWWFG5J4OSO36WRY7J6KU5UBCKYSBJMSJDKP7HHAXREVQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.987456879 (198h16m46.683305632s) [noise=true , address=77.90.31.11:4554 , piece_id=VBPMMLIAQORHZN2HZOW2GOX357NCZOOJQXI6EAUBTLGZTCQXLBPA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.987835771 (198h16m46.683684524s) [noise=true , address=145.79.227.31:4799 , piece_id=KGI3O3SRIMFEE6A2XBBXIDT5DKJ2AARHT42QOIZHWNEQKVHDPEFQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.988197610 (198h16m46.684046363s) [noise=true , address=145.79.227.30:4763 , piece_id=5NULCCW3JQN4BJRULGWNZYTLMMNHFWVBWTHM5I6YCOC3DSXLM3YA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.988573415 (198h16m46.684422168s) [noise=true , address=77.90.31.17:4771 , piece_id=LIPQS5ZA6FIJPVTILV7KR7IVPEKOSCHFS6DYKYXFRZBNMGZNF5OQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.988966293 (198h16m46.684815046s) [noise=true , address=145.79.227.27:4689 , piece_id=BXARKOIISINDW3VTRJEJCTFFRTMOAWF6FLKXQM6WETANR35FDASQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.989256176 (198h16m46.685104929s) [noise=true , address=77.90.31.18:4807 , piece_id=Q33WEQIRN5YKFMY2TXWPI7NPJKVAH5IS4VEFXQ6GWCPP5C56TL4A, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.989762870 (198h16m46.685611623s) [noise=true , address=145.79.227.23:4547 , piece_id=2VIEJA6UFC2GR4NE47KWMZSGGK635FEXASMZV7XEDL4CDSHIW6BA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.990398070 (198h16m46.686246824s) [noise=true , address=77.90.31.12:4589 , piece_id=2SRMOX47TVHH2KECGPSYGUDO6NKOKK6XZPMUHNTA4DF3A6MUKC5Q, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.990770419 (198h16m46.686619172s) [noise=true , address=145.79.227.14:4839 , piece_id=SV7UM6CRURIZGJK4HNYK57F2H2DCX5T5YCHS6IURTUKF2W6O2I2Q, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.991225625 (198h16m46.687074369s) [noise=true , address=45.91.116.14:4190 , piece_id=K2OQ3PERPEBMOSHCGADQBRW2GA3NDWBQJHSGDX5E2KZ37IKTZ5MA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.993159528 (198h16m46.689008281s) [noise=true , address=45.91.116.8:4048 , piece_id=7X2CWYKYQM2AYP4S43H6XQTELRZZSNJAXKFWI6HHTQG7PY2OKAZQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.994399059 (198h16m46.690247802s) [noise=true , address=145.79.227.32:4836 , piece_id=7YY2EIPXSIQVAGCLSAW6ZAXOHKE3AAGCDTHQEW33E4NO37XH6UDA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.995647857 (198h16m46.69149661s) [noise=true , address=145.79.227.9:4366 , piece_id=4ML6LCVTLOFV3DED4PJEERC7HHD2H7RZJI5TFRSCXFKHLW5LMCXQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.996969854 (198h16m46.692818598s) [noise=true , address=45.91.116.6:4975 , piece_id=BWRFVPHVZLEUMQINBXTX4HFZKPUPLTU6G4B5EMV36GMZ2PNXIBBA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:16.764811705 (198h16m48.460660448s) [upload=174748]: Next returned nil segment Nov 7 17:57:16.764837084 (198h16m48.460685827s) [upload=174748]: Waiting for error group managing segments... Nov 7 17:57:17.048338734 (198h16m48.744187487s) [noise=true , address=145.79.227.14:4839 , piece_id=SV7UM6CRURIZGJK4HNYK57F2H2DCX5T5YCHS6IURTUKF2W6O2I2Q, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.055321240 (198h16m48.751169983s) [noise=true , address=145.79.227.23:4547 , piece_id=2VIEJA6UFC2GR4NE47KWMZSGGK635FEXASMZV7XEDL4CDSHIW6BA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.105778392 (198h16m48.801627145s) [noise=true , address=145.79.227.32:4836 , piece_id=7YY2EIPXSIQVAGCLSAW6ZAXOHKE3AAGCDTHQEW33E4NO37XH6UDA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.107698748 (198h16m48.803547501s) [noise=true , address=145.79.227.27:4689 , piece_id=BXARKOIISINDW3VTRJEJCTFFRTMOAWF6FLKXQM6WETANR35FDASQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.151234479 (198h16m48.847083232s) [noise=true , address=145.79.227.9:4366 , piece_id=4ML6LCVTLOFV3DED4PJEERC7HHD2H7RZJI5TFRSCXFKHLW5LMCXQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.188892929 (198h16m48.884741682s) [noise=true , address=145.79.227.31:4799 , piece_id=KGI3O3SRIMFEE6A2XBBXIDT5DKJ2AARHT42QOIZHWNEQKVHDPEFQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.204681464 (198h16m48.900530217s) [noise=true , address=145.79.227.30:4763 , piece_id=5NULCCW3JQN4BJRULGWNZYTLMMNHFWVBWTHM5I6YCOC3DSXLM3YA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.227299704 (198h16m48.923148457s) [noise=true , address=45.91.116.6:4975 , piece_id=BWRFVPHVZLEUMQINBXTX4HFZKPUPLTU6G4B5EMV36GMZ2PNXIBBA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.240346959 (198h16m48.936195742s) [noise=true , address=77.90.31.17:4771 , piece_id=LIPQS5ZA6FIJPVTILV7KR7IVPEKOSCHFS6DYKYXFRZBNMGZNF5OQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.252076615 (198h16m48.947925368s) [noise=true , address=45.91.116.8:4048 , piece_id=7X2CWYKYQM2AYP4S43H6XQTELRZZSNJAXKFWI6HHTQG7PY2OKAZQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.264161628 (198h16m48.960010371s) [noise=true , address=77.90.31.11:4554 , piece_id=VBPMMLIAQORHZN2HZOW2GOX357NCZOOJQXI6EAUBTLGZTCQXLBPA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.268966687 (198h16m48.96481544s) [noise=true , address=77.90.31.12:4589 , piece_id=2SRMOX47TVHH2KECGPSYGUDO6NKOKK6XZPMUHNTA4DF3A6MUKC5Q, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.278743855 (198h16m48.974592598s) [noise=true , address=45.91.116.14:4190 , piece_id=K2OQ3PERPEBMOSHCGADQBRW2GA3NDWBQJHSGDX5E2KZ37IKTZ5MA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.278767590 (198h16m48.974616343s) [seg_pos={0 0}, upload=174748]: Segment reached optimal threshold of 13 pieces. Nov 7 17:57:17.279090265 (198h16m48.974939018s) [noise=true , address=77.90.31.18:4807 , piece_id=Q33WEQIRN5YKFMY2TXWPI7NPJKVAH5IS4VEFXQ6GWCPP5C56TL4A, seg_pos={0 0}, upload=174748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Nov 7 17:57:17.279097098 (198h16m48.974945841s) [noise=true , address=77.90.31.18:4807 , piece_id=Q33WEQIRN5YKFMY2TXWPI7NPJKVAH5IS4VEFXQ6GWCPP5C56TL4A, seg_pos={0 0}, upload=174748]: Long tail cancellation due to optimal threshold Nov 7 17:57:17.279275107 (198h16m48.97512387s) [noise=true , address=77.90.31.12:4590 , piece_id=POZAAVNGWWFG5J4OSO36WRY7J6KU5UBCKYSBJMSJDKP7HHAXREVQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Nov 7 17:57:17.279296748 (198h16m48.975145501s) [noise=true , address=77.90.31.12:4590 , piece_id=POZAAVNGWWFG5J4OSO36WRY7J6KU5UBCKYSBJMSJDKP7HHAXREVQ, seg_pos={0 0}, upload=174748]: Long tail cancellation due to optimal threshold Nov 7 17:57:17.279385046 (198h16m48.975233799s) [noise=true , address=77.90.31.21:4879 , piece_id=PU77CJBPJOFVVWFA52ZM5GZIZASKUWDYS6P3L44IZNJ2LHDPYBSA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 7 17:57:17.279393522 (198h16m48.975242275s) [noise=true , address=77.90.31.21:4879 , piece_id=PU77CJBPJOFVVWFA52ZM5GZIZASKUWDYS6P3L44IZNJ2LHDPYBSA, seg_pos={0 0}, upload=174748]: Long tail cancellation due to optimal threshold Nov 7 17:57:17.279411617 (198h16m48.97526037s) [seg_pos={0 0}, upload=174748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:57:17.279495146 (198h16m48.975343889s) [upload=174748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:57:17.279501638 (198h16m48.975350391s) [upload=174748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:57:17.577861542 (198h16m49.273710315s) [upload=174748]: Done uploading segments... Nov 7 17:57:17.577899043 (198h16m49.273747796s) [upload=174748]: Upload finished. err: <nil> Duration: 7.679787896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.426698266s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:58:11 UTC) orbiter warning: context deadline exceeded Duration: 3.426731148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X] duration: 30.003196802s trace: https://jaeger.dev.storj.io/trace/2520db66a1bfb6d6 FAIL (2025-11-07 17:57:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 7 17:57:14.739375879 (198h16m46.435224632s) [upload=174748]: Starting upload Nov 7 17:57:14.739462223 (198h16m46.435310976s) [upload=174748]: Done starting upload Nov 7 17:57:14.739687112 (198h16m46.435535865s) [upload=174748]: Uploading segments... Nov 7 17:57:14.739763296 (198h16m46.43561205s) [upload=174748]: Got next segment. Inline: false Nov 7 17:57:14.739806308 (198h16m46.435655061s) [upload=174748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:57:14.739813603 (198h16m46.435662366s) [upload=174748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:57:14.986182943 (198h16m46.682031696s) [seg_pos={0 0}, upload=174748]: Begin upload segment... Nov 7 17:57:14.986240203 (198h16m46.682088966s) [seg_pos={0 0}, upload=174748]: Done begin upload segment. Nov 7 17:57:14.986343770 (198h16m46.682192553s) [noise=true , address=77.90.31.21:4879 , piece_id=PU77CJBPJOFVVWFA52ZM5GZIZASKUWDYS6P3L44IZNJ2LHDPYBSA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.986975212 (198h16m46.682823965s) [noise=true , address=77.90.31.12:4590 , piece_id=POZAAVNGWWFG5J4OSO36WRY7J6KU5UBCKYSBJMSJDKP7HHAXREVQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.987456879 (198h16m46.683305632s) [noise=true , address=77.90.31.11:4554 , piece_id=VBPMMLIAQORHZN2HZOW2GOX357NCZOOJQXI6EAUBTLGZTCQXLBPA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.987835771 (198h16m46.683684524s) [noise=true , address=145.79.227.31:4799 , piece_id=KGI3O3SRIMFEE6A2XBBXIDT5DKJ2AARHT42QOIZHWNEQKVHDPEFQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.988197610 (198h16m46.684046363s) [noise=true , address=145.79.227.30:4763 , piece_id=5NULCCW3JQN4BJRULGWNZYTLMMNHFWVBWTHM5I6YCOC3DSXLM3YA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.988573415 (198h16m46.684422168s) [noise=true , address=77.90.31.17:4771 , piece_id=LIPQS5ZA6FIJPVTILV7KR7IVPEKOSCHFS6DYKYXFRZBNMGZNF5OQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.988966293 (198h16m46.684815046s) [noise=true , address=145.79.227.27:4689 , piece_id=BXARKOIISINDW3VTRJEJCTFFRTMOAWF6FLKXQM6WETANR35FDASQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.989256176 (198h16m46.685104929s) [noise=true , address=77.90.31.18:4807 , piece_id=Q33WEQIRN5YKFMY2TXWPI7NPJKVAH5IS4VEFXQ6GWCPP5C56TL4A, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.989762870 (198h16m46.685611623s) [noise=true , address=145.79.227.23:4547 , piece_id=2VIEJA6UFC2GR4NE47KWMZSGGK635FEXASMZV7XEDL4CDSHIW6BA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.990398070 (198h16m46.686246824s) [noise=true , address=77.90.31.12:4589 , piece_id=2SRMOX47TVHH2KECGPSYGUDO6NKOKK6XZPMUHNTA4DF3A6MUKC5Q, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.990770419 (198h16m46.686619172s) [noise=true , address=145.79.227.14:4839 , piece_id=SV7UM6CRURIZGJK4HNYK57F2H2DCX5T5YCHS6IURTUKF2W6O2I2Q, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.991225625 (198h16m46.687074369s) [noise=true , address=45.91.116.14:4190 , piece_id=K2OQ3PERPEBMOSHCGADQBRW2GA3NDWBQJHSGDX5E2KZ37IKTZ5MA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.993159528 (198h16m46.689008281s) [noise=true , address=45.91.116.8:4048 , piece_id=7X2CWYKYQM2AYP4S43H6XQTELRZZSNJAXKFWI6HHTQG7PY2OKAZQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.994399059 (198h16m46.690247802s) [noise=true , address=145.79.227.32:4836 , piece_id=7YY2EIPXSIQVAGCLSAW6ZAXOHKE3AAGCDTHQEW33E4NO37XH6UDA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.995647857 (198h16m46.69149661s) [noise=true , address=145.79.227.9:4366 , piece_id=4ML6LCVTLOFV3DED4PJEERC7HHD2H7RZJI5TFRSCXFKHLW5LMCXQ, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:14.996969854 (198h16m46.692818598s) [noise=true , address=45.91.116.6:4975 , piece_id=BWRFVPHVZLEUMQINBXTX4HFZKPUPLTU6G4B5EMV36GMZ2PNXIBBA, seg_pos={0 0}, upload=174748]: Uploading piece... Nov 7 17:57:16.764811705 (198h16m48.460660448s) [upload=174748]: Next returned nil segment Nov 7 17:57:16.764837084 (198h16m48.460685827s) [upload=174748]: Waiting for error group managing segments... Nov 7 17:57:17.048338734 (198h16m48.744187487s) [noise=true , address=145.79.227.14:4839 , piece_id=SV7UM6CRURIZGJK4HNYK57F2H2DCX5T5YCHS6IURTUKF2W6O2I2Q, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.055321240 (198h16m48.751169983s) [noise=true , address=145.79.227.23:4547 , piece_id=2VIEJA6UFC2GR4NE47KWMZSGGK635FEXASMZV7XEDL4CDSHIW6BA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.105778392 (198h16m48.801627145s) [noise=true , address=145.79.227.32:4836 , piece_id=7YY2EIPXSIQVAGCLSAW6ZAXOHKE3AAGCDTHQEW33E4NO37XH6UDA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.107698748 (198h16m48.803547501s) [noise=true , address=145.79.227.27:4689 , piece_id=BXARKOIISINDW3VTRJEJCTFFRTMOAWF6FLKXQM6WETANR35FDASQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.151234479 (198h16m48.847083232s) [noise=true , address=145.79.227.9:4366 , piece_id=4ML6LCVTLOFV3DED4PJEERC7HHD2H7RZJI5TFRSCXFKHLW5LMCXQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.188892929 (198h16m48.884741682s) [noise=true , address=145.79.227.31:4799 , piece_id=KGI3O3SRIMFEE6A2XBBXIDT5DKJ2AARHT42QOIZHWNEQKVHDPEFQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.204681464 (198h16m48.900530217s) [noise=true , address=145.79.227.30:4763 , piece_id=5NULCCW3JQN4BJRULGWNZYTLMMNHFWVBWTHM5I6YCOC3DSXLM3YA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.227299704 (198h16m48.923148457s) [noise=true , address=45.91.116.6:4975 , piece_id=BWRFVPHVZLEUMQINBXTX4HFZKPUPLTU6G4B5EMV36GMZ2PNXIBBA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.240346959 (198h16m48.936195742s) [noise=true , address=77.90.31.17:4771 , piece_id=LIPQS5ZA6FIJPVTILV7KR7IVPEKOSCHFS6DYKYXFRZBNMGZNF5OQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.252076615 (198h16m48.947925368s) [noise=true , address=45.91.116.8:4048 , piece_id=7X2CWYKYQM2AYP4S43H6XQTELRZZSNJAXKFWI6HHTQG7PY2OKAZQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.264161628 (198h16m48.960010371s) [noise=true , address=77.90.31.11:4554 , piece_id=VBPMMLIAQORHZN2HZOW2GOX357NCZOOJQXI6EAUBTLGZTCQXLBPA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.268966687 (198h16m48.96481544s) [noise=true , address=77.90.31.12:4589 , piece_id=2SRMOX47TVHH2KECGPSYGUDO6NKOKK6XZPMUHNTA4DF3A6MUKC5Q, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.278743855 (198h16m48.974592598s) [noise=true , address=45.91.116.14:4190 , piece_id=K2OQ3PERPEBMOSHCGADQBRW2GA3NDWBQJHSGDX5E2KZ37IKTZ5MA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: <nil> Nov 7 17:57:17.278767590 (198h16m48.974616343s) [seg_pos={0 0}, upload=174748]: Segment reached optimal threshold of 13 pieces. Nov 7 17:57:17.279090265 (198h16m48.974939018s) [noise=true , address=77.90.31.18:4807 , piece_id=Q33WEQIRN5YKFMY2TXWPI7NPJKVAH5IS4VEFXQ6GWCPP5C56TL4A, seg_pos={0 0}, upload=174748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Nov 7 17:57:17.279097098 (198h16m48.974945841s) [noise=true , address=77.90.31.18:4807 , piece_id=Q33WEQIRN5YKFMY2TXWPI7NPJKVAH5IS4VEFXQ6GWCPP5C56TL4A, seg_pos={0 0}, upload=174748]: Long tail cancellation due to optimal threshold Nov 7 17:57:17.279275107 (198h16m48.97512387s) [noise=true , address=77.90.31.12:4590 , piece_id=POZAAVNGWWFG5J4OSO36WRY7J6KU5UBCKYSBJMSJDKP7HHAXREVQ, seg_pos={0 0}, upload=174748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Nov 7 17:57:17.279296748 (198h16m48.975145501s) [noise=true , address=77.90.31.12:4590 , piece_id=POZAAVNGWWFG5J4OSO36WRY7J6KU5UBCKYSBJMSJDKP7HHAXREVQ, seg_pos={0 0}, upload=174748]: Long tail cancellation due to optimal threshold Nov 7 17:57:17.279385046 (198h16m48.975233799s) [noise=true , address=77.90.31.21:4879 , piece_id=PU77CJBPJOFVVWFA52ZM5GZIZASKUWDYS6P3L44IZNJ2LHDPYBSA, seg_pos={0 0}, upload=174748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 7 17:57:17.279393522 (198h16m48.975242275s) [noise=true , address=77.90.31.21:4879 , piece_id=PU77CJBPJOFVVWFA52ZM5GZIZASKUWDYS6P3L44IZNJ2LHDPYBSA, seg_pos={0 0}, upload=174748]: Long tail cancellation due to optimal threshold Nov 7 17:57:17.279411617 (198h16m48.97526037s) [seg_pos={0 0}, upload=174748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:57:17.279495146 (198h16m48.975343889s) [upload=174748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:57:17.279501638 (198h16m48.975350391s) [upload=174748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:57:17.577861542 (198h16m49.273710315s) [upload=174748]: Done uploading segments... Nov 7 17:57:17.577899043 (198h16m49.273747796s) [upload=174748]: Upload finished. err: <nil> Duration: 30.003225837s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.61225655s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:57:30 UTC) orbiter warning: context deadline exceeded Duration: 3.612287761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.632851057s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:56:45 UTC) orbiter warning: context deadline exceeded Duration: 3.632880714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.] duration: 6.781603253s trace: https://jaeger.dev.storj.io/trace/44c1e00c089719d0 FAIL (2025-11-07 17:56:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7:54:59.368298019 (198h14m31.064146762s) [upload=174716]: Starting upload Nov 7 17:54:59.368368173 (198h14m31.064216926s) [upload=174716]: Done starting upload Nov 7 17:54:59.368461411 (198h14m31.064310174s) [upload=174716]: Uploading segments... Nov 7 17:54:59.368562112 (198h14m31.064410865s) [upload=174716]: Got next segment. Inline: false Nov 7 17:54:59.368604623 (198h14m31.064453376s) [upload=174716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:54:59.368610915 (198h14m31.064459668s) [upload=174716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:54:59.623816553 (198h14m31.319665306s) [seg_pos={0 0}, upload=174716]: Begin upload segment... Nov 7 17:54:59.623890093 (198h14m31.319738847s) [seg_pos={0 0}, upload=174716]: Done begin upload segment. Nov 7 17:54:59.623945669 (198h14m31.319794422s) [noise=true , address=145.79.227.13:4803 , piece_id=7D4KZK3ZBM5XWIBYGOTY4CVS5Q6O6KPSDDXGQASS3HP5HJKLOYGA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.624073773 (198h14m31.319922526s) [noise=true , address=45.91.116.11:4118 , piece_id=NTWG66L6XXMFL2IHN5GMEKY6TWYXY5AI2JZLFOYKZFESQ6E6GGZA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.624709243 (198h14m31.320557996s) [noise=true , address=77.90.31.10:4517 , piece_id=QU3P6OXHRSQTMLBTCSGPGD227CV5TP3NQ6VPZDDXVE3OSNRRRAPQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.625144542 (198h14m31.320993295s) [noise=true , address=145.79.227.31:4799 , piece_id=22NY5MQFE7WGUNMROELNIGUUEPWODL2Z75HMQPLFTIKVABCSM4BQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.625565544 (198h14m31.321414297s) [noise=true , address=145.79.227.14:4840 , piece_id=J2Q2PRNUIHHQRY5UU72DBOKVNU62DZ7IE7UWITWBP376KWDW2PEQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.626085875 (198h14m31.321934628s) [noise=true , address=145.79.227.20:4439 , piece_id=CN6ZY5ZXETTHPPHDYQP7D2HG2SHMITOWXPPZGBWF5V4BYIKCWNTA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.626204550 (198h14m31.322053303s) [noise=true , address=145.79.227.15:4874 , piece_id=T222JHJ2HS6YXQ2X6DSWFOHKPKH7ZSKTOZBBTOIVH4ZS2H4FAXFA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.627613303 (198h14m31.323462056s) [noise=true , address=145.79.227.17:4971 , piece_id=KQIN65UEX55TU5URO74YWH6K7EMRNM3FXPG6L2LDAEVEUVXO62UQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.627807442 (198h14m31.323656195s) [noise=true , address=145.79.227.4:4186 , piece_id=N746S5A2JWPD3KSEHRX2AB6VVKWCGDN33OPZV5NK3BMGGRQRAQXQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.629169466 (198h14m31.325018219s) [noise=true , address=145.79.227.26:4654 , piece_id=BUPGNIISKQ24OYQK75D6J4LRP2RODM2K26XXP2ZSIYGWAM45WMUQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.629462885 (198h14m31.325311638s) [noise=true , address=77.90.31.21:4879 , piece_id=UADU4IUQ2KQ2EFRVMF6FWQ4HZNRBAJ7U3YKAR6D4C3VUKKQIFLOA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.630885494 (198h14m31.326734247s) [noise=true , address=145.79.227.32:4835 , piece_id=5EXVL5CJ6YXAKE6M3J6L5Z7QM3MCDC3MY5PSVZ25F43OBTAM47LA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.632542188 (198h14m31.328390941s) [noise=true , address=45.91.116.14:4189 , piece_id=NK7BSDAECL2XNKYVYNCGTN5RAS573ROETVOLAFSR26R3GB55V6SQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.633919942 (198h14m31.329768695s) [noise=true , address=145.79.227.9:4365 , piece_id=B6HH2HXWDKTVK5U3K4JLHQ5BWDPWTGEA3N66GX2MZO54PNL7VOPA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.634850765 (198h14m31.330699518s) [noise=true , address=45.91.116.12:4226 , piece_id=DSK4UFRUJY2RQUJRL62LVB3BX4HTNEJOQJ6ECBLGKYUD2EWJRB7A, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.636580018 (198h14m31.332428771s) [noise=true , address=145.79.227.11:4732 , piece_id=RWWT7WBQ2SFC2YCKMNHS2QFHEDVXGC5ME3UCGYTRJ5BQTUB2UYQQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:55:01.360570394 (198h14m33.056419157s) [upload=174716]: Next returned nil segment Nov 7 17:55:01.360591203 (198h14m33.056439956s) [upload=174716]: Waiting for error group managing segments... Nov 7 17:55:01.708595801 (198h14m33.404444554s) [noise=true , address=145.79.227.31:4799 , piece_id=22NY5MQFE7WGUNMROELNIGUUEPWODL2Z75HMQPLFTIKVABCSM4BQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.733484915 (198h14m33.429333658s) [noise=true , address=145.79.227.14:4840 , piece_id=J2Q2PRNUIHHQRY5UU72DBOKVNU62DZ7IE7UWITWBP376KWDW2PEQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.742188259 (198h14m33.438037022s) [noise=true , address=145.79.227.9:4365 , piece_id=B6HH2HXWDKTVK5U3K4JLHQ5BWDPWTGEA3N66GX2MZO54PNL7VOPA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.743183514 (198h14m33.439032267s) [noise=true , address=145.79.227.32:4835 , piece_id=5EXVL5CJ6YXAKE6M3J6L5Z7QM3MCDC3MY5PSVZ25F43OBTAM47LA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.750760742 (198h14m33.446609495s) [noise=true , address=145.79.227.11:4732 , piece_id=RWWT7WBQ2SFC2YCKMNHS2QFHEDVXGC5ME3UCGYTRJ5BQTUB2UYQQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.756508598 (198h14m33.452357351s) [noise=true , address=145.79.227.26:4654 , piece_id=BUPGNIISKQ24OYQK75D6J4LRP2RODM2K26XXP2ZSIYGWAM45WMUQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.764872304 (198h14m33.460721058s) [noise=true , address=145.79.227.17:4971 , piece_id=KQIN65UEX55TU5URO74YWH6K7EMRNM3FXPG6L2LDAEVEUVXO62UQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.767767437 (198h14m33.46361619s) [noise=true , address=145.79.227.15:4874 , piece_id=T222JHJ2HS6YXQ2X6DSWFOHKPKH7ZSKTOZBBTOIVH4ZS2H4FAXFA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.804059917 (198h14m33.49990868s) [noise=true , address=145.79.227.20:4439 , piece_id=CN6ZY5ZXETTHPPHDYQP7D2HG2SHMITOWXPPZGBWF5V4BYIKCWNTA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.812609808 (198h14m33.508458561s) [noise=true , address=145.79.227.13:4803 , piece_id=7D4KZK3ZBM5XWIBYGOTY4CVS5Q6O6KPSDDXGQASS3HP5HJKLOYGA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.879596745 (198h14m33.575445498s) [noise=true , address=145.79.227.4:4186 , piece_id=N746S5A2JWPD3KSEHRX2AB6VVKWCGDN33OPZV5NK3BMGGRQRAQXQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:02.020058495 (198h14m33.715907258s) [noise=true , address=77.90.31.10:4517 , piece_id=QU3P6OXHRSQTMLBTCSGPGD227CV5TP3NQ6VPZDDXVE3OSNRRRAPQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:02.654992340 (198h14m34.350841093s) [noise=true , address=77.90.31.21:4879 , piece_id=UADU4IUQ2KQ2EFRVMF6FWQ4HZNRBAJ7U3YKAR6D4C3VUKKQIFLOA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:02.655019471 (198h14m34.350868224s) [seg_pos={0 0}, upload=174716]: Segment reached optimal threshold of 13 pieces. Nov 7 17:55:02.655681822 (198h14m34.351530577s) [noise=true , address=45.91.116.14:4189 , piece_id=NK7BSDAECL2XNKYVYNCGTN5RAS573ROETVOLAFSR26R3GB55V6SQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 17:55:02.655693726 (198h14m34.351542479s) [noise=true , address=45.91.116.14:4189 , piece_id=NK7BSDAECL2XNKYVYNCGTN5RAS573ROETVOLAFSR26R3GB55V6SQ, seg_pos={0 0}, upload=174716]: Long tail cancellation due to optimal threshold Nov 7 17:55:02.658474521 (198h14m34.354323274s) [noise=true , address=45.91.116.12:4226 , piece_id=DSK4UFRUJY2RQUJRL62LVB3BX4HTNEJOQJ6ECBLGKYUD2EWJRB7A, seg_pos={0 0}, upload=174716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Nov 7 17:55:02.658488157 (198h14m34.35433691s) [noise=true , address=45.91.116.12:4226 , piece_id=DSK4UFRUJY2RQUJRL62LVB3BX4HTNEJOQJ6ECBLGKYUD2EWJRB7A, seg_pos={0 0}, upload=174716]: Long tail cancellation due to optimal threshold Nov 7 17:55:02.658734866 (198h14m34.354583619s) [noise=true , address=45.91.116.11:4118 , piece_id=NTWG66L6XXMFL2IHN5GMEKY6TWYXY5AI2JZLFOYKZFESQ6E6GGZA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:51360->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:51360->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51360->45.91.116.11:4118: use of closed network connection Nov 7 17:55:02.658774401 (198h14m34.354623154s) [noise=true , address=45.91.116.11:4118 , piece_id=NTWG66L6XXMFL2IHN5GMEKY6TWYXY5AI2JZLFOYKZFESQ6E6GGZA, seg_pos={0 0}, upload=174716]: Long tail cancellation due to optimal threshold Nov 7 17:55:02.658800281 (198h14m34.354649034s) [seg_pos={0 0}, upload=174716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:55:02.659857233 (198h14m34.355705986s) [upload=174716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:55:02.659867564 (198h14m34.355716317s) [upload=174716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:55:02.975488404 (198h14m34.671337157s) [upload=174716]: Done uploading segments... Nov 7 17:55:02.975541576 (198h14m34.671390329s) [upload=174716]: Upload finished. err: <nil> Duration: 6.781617761s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.580852051s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:56:00 UTC) orbiter warning: context deadline exceeded Duration: 3.580881446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X] duration: 30.007189547s trace: https://jaeger.dev.storj.io/trace/44c1e00c089719d0 FAIL (2025-11-07 17:55:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7:54:59.368298019 (198h14m31.064146762s) [upload=174716]: Starting upload Nov 7 17:54:59.368368173 (198h14m31.064216926s) [upload=174716]: Done starting upload Nov 7 17:54:59.368461411 (198h14m31.064310174s) [upload=174716]: Uploading segments... Nov 7 17:54:59.368562112 (198h14m31.064410865s) [upload=174716]: Got next segment. Inline: false Nov 7 17:54:59.368604623 (198h14m31.064453376s) [upload=174716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:54:59.368610915 (198h14m31.064459668s) [upload=174716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:54:59.623816553 (198h14m31.319665306s) [seg_pos={0 0}, upload=174716]: Begin upload segment... Nov 7 17:54:59.623890093 (198h14m31.319738847s) [seg_pos={0 0}, upload=174716]: Done begin upload segment. Nov 7 17:54:59.623945669 (198h14m31.319794422s) [noise=true , address=145.79.227.13:4803 , piece_id=7D4KZK3ZBM5XWIBYGOTY4CVS5Q6O6KPSDDXGQASS3HP5HJKLOYGA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.624073773 (198h14m31.319922526s) [noise=true , address=45.91.116.11:4118 , piece_id=NTWG66L6XXMFL2IHN5GMEKY6TWYXY5AI2JZLFOYKZFESQ6E6GGZA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.624709243 (198h14m31.320557996s) [noise=true , address=77.90.31.10:4517 , piece_id=QU3P6OXHRSQTMLBTCSGPGD227CV5TP3NQ6VPZDDXVE3OSNRRRAPQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.625144542 (198h14m31.320993295s) [noise=true , address=145.79.227.31:4799 , piece_id=22NY5MQFE7WGUNMROELNIGUUEPWODL2Z75HMQPLFTIKVABCSM4BQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.625565544 (198h14m31.321414297s) [noise=true , address=145.79.227.14:4840 , piece_id=J2Q2PRNUIHHQRY5UU72DBOKVNU62DZ7IE7UWITWBP376KWDW2PEQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.626085875 (198h14m31.321934628s) [noise=true , address=145.79.227.20:4439 , piece_id=CN6ZY5ZXETTHPPHDYQP7D2HG2SHMITOWXPPZGBWF5V4BYIKCWNTA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.626204550 (198h14m31.322053303s) [noise=true , address=145.79.227.15:4874 , piece_id=T222JHJ2HS6YXQ2X6DSWFOHKPKH7ZSKTOZBBTOIVH4ZS2H4FAXFA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.627613303 (198h14m31.323462056s) [noise=true , address=145.79.227.17:4971 , piece_id=KQIN65UEX55TU5URO74YWH6K7EMRNM3FXPG6L2LDAEVEUVXO62UQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.627807442 (198h14m31.323656195s) [noise=true , address=145.79.227.4:4186 , piece_id=N746S5A2JWPD3KSEHRX2AB6VVKWCGDN33OPZV5NK3BMGGRQRAQXQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.629169466 (198h14m31.325018219s) [noise=true , address=145.79.227.26:4654 , piece_id=BUPGNIISKQ24OYQK75D6J4LRP2RODM2K26XXP2ZSIYGWAM45WMUQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.629462885 (198h14m31.325311638s) [noise=true , address=77.90.31.21:4879 , piece_id=UADU4IUQ2KQ2EFRVMF6FWQ4HZNRBAJ7U3YKAR6D4C3VUKKQIFLOA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.630885494 (198h14m31.326734247s) [noise=true , address=145.79.227.32:4835 , piece_id=5EXVL5CJ6YXAKE6M3J6L5Z7QM3MCDC3MY5PSVZ25F43OBTAM47LA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.632542188 (198h14m31.328390941s) [noise=true , address=45.91.116.14:4189 , piece_id=NK7BSDAECL2XNKYVYNCGTN5RAS573ROETVOLAFSR26R3GB55V6SQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.633919942 (198h14m31.329768695s) [noise=true , address=145.79.227.9:4365 , piece_id=B6HH2HXWDKTVK5U3K4JLHQ5BWDPWTGEA3N66GX2MZO54PNL7VOPA, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.634850765 (198h14m31.330699518s) [noise=true , address=45.91.116.12:4226 , piece_id=DSK4UFRUJY2RQUJRL62LVB3BX4HTNEJOQJ6ECBLGKYUD2EWJRB7A, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:54:59.636580018 (198h14m31.332428771s) [noise=true , address=145.79.227.11:4732 , piece_id=RWWT7WBQ2SFC2YCKMNHS2QFHEDVXGC5ME3UCGYTRJ5BQTUB2UYQQ, seg_pos={0 0}, upload=174716]: Uploading piece... Nov 7 17:55:01.360570394 (198h14m33.056419157s) [upload=174716]: Next returned nil segment Nov 7 17:55:01.360591203 (198h14m33.056439956s) [upload=174716]: Waiting for error group managing segments... Nov 7 17:55:01.708595801 (198h14m33.404444554s) [noise=true , address=145.79.227.31:4799 , piece_id=22NY5MQFE7WGUNMROELNIGUUEPWODL2Z75HMQPLFTIKVABCSM4BQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.733484915 (198h14m33.429333658s) [noise=true , address=145.79.227.14:4840 , piece_id=J2Q2PRNUIHHQRY5UU72DBOKVNU62DZ7IE7UWITWBP376KWDW2PEQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.742188259 (198h14m33.438037022s) [noise=true , address=145.79.227.9:4365 , piece_id=B6HH2HXWDKTVK5U3K4JLHQ5BWDPWTGEA3N66GX2MZO54PNL7VOPA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.743183514 (198h14m33.439032267s) [noise=true , address=145.79.227.32:4835 , piece_id=5EXVL5CJ6YXAKE6M3J6L5Z7QM3MCDC3MY5PSVZ25F43OBTAM47LA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.750760742 (198h14m33.446609495s) [noise=true , address=145.79.227.11:4732 , piece_id=RWWT7WBQ2SFC2YCKMNHS2QFHEDVXGC5ME3UCGYTRJ5BQTUB2UYQQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.756508598 (198h14m33.452357351s) [noise=true , address=145.79.227.26:4654 , piece_id=BUPGNIISKQ24OYQK75D6J4LRP2RODM2K26XXP2ZSIYGWAM45WMUQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.764872304 (198h14m33.460721058s) [noise=true , address=145.79.227.17:4971 , piece_id=KQIN65UEX55TU5URO74YWH6K7EMRNM3FXPG6L2LDAEVEUVXO62UQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.767767437 (198h14m33.46361619s) [noise=true , address=145.79.227.15:4874 , piece_id=T222JHJ2HS6YXQ2X6DSWFOHKPKH7ZSKTOZBBTOIVH4ZS2H4FAXFA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.804059917 (198h14m33.49990868s) [noise=true , address=145.79.227.20:4439 , piece_id=CN6ZY5ZXETTHPPHDYQP7D2HG2SHMITOWXPPZGBWF5V4BYIKCWNTA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.812609808 (198h14m33.508458561s) [noise=true , address=145.79.227.13:4803 , piece_id=7D4KZK3ZBM5XWIBYGOTY4CVS5Q6O6KPSDDXGQASS3HP5HJKLOYGA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:01.879596745 (198h14m33.575445498s) [noise=true , address=145.79.227.4:4186 , piece_id=N746S5A2JWPD3KSEHRX2AB6VVKWCGDN33OPZV5NK3BMGGRQRAQXQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:02.020058495 (198h14m33.715907258s) [noise=true , address=77.90.31.10:4517 , piece_id=QU3P6OXHRSQTMLBTCSGPGD227CV5TP3NQ6VPZDDXVE3OSNRRRAPQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:02.654992340 (198h14m34.350841093s) [noise=true , address=77.90.31.21:4879 , piece_id=UADU4IUQ2KQ2EFRVMF6FWQ4HZNRBAJ7U3YKAR6D4C3VUKKQIFLOA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: <nil> Nov 7 17:55:02.655019471 (198h14m34.350868224s) [seg_pos={0 0}, upload=174716]: Segment reached optimal threshold of 13 pieces. Nov 7 17:55:02.655681822 (198h14m34.351530577s) [noise=true , address=45.91.116.14:4189 , piece_id=NK7BSDAECL2XNKYVYNCGTN5RAS573ROETVOLAFSR26R3GB55V6SQ, seg_pos={0 0}, upload=174716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 17:55:02.655693726 (198h14m34.351542479s) [noise=true , address=45.91.116.14:4189 , piece_id=NK7BSDAECL2XNKYVYNCGTN5RAS573ROETVOLAFSR26R3GB55V6SQ, seg_pos={0 0}, upload=174716]: Long tail cancellation due to optimal threshold Nov 7 17:55:02.658474521 (198h14m34.354323274s) [noise=true , address=45.91.116.12:4226 , piece_id=DSK4UFRUJY2RQUJRL62LVB3BX4HTNEJOQJ6ECBLGKYUD2EWJRB7A, seg_pos={0 0}, upload=174716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Nov 7 17:55:02.658488157 (198h14m34.35433691s) [noise=true , address=45.91.116.12:4226 , piece_id=DSK4UFRUJY2RQUJRL62LVB3BX4HTNEJOQJ6ECBLGKYUD2EWJRB7A, seg_pos={0 0}, upload=174716]: Long tail cancellation due to optimal threshold Nov 7 17:55:02.658734866 (198h14m34.354583619s) [noise=true , address=45.91.116.11:4118 , piece_id=NTWG66L6XXMFL2IHN5GMEKY6TWYXY5AI2JZLFOYKZFESQ6E6GGZA, seg_pos={0 0}, upload=174716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:51360->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:51360->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51360->45.91.116.11:4118: use of closed network connection Nov 7 17:55:02.658774401 (198h14m34.354623154s) [noise=true , address=45.91.116.11:4118 , piece_id=NTWG66L6XXMFL2IHN5GMEKY6TWYXY5AI2JZLFOYKZFESQ6E6GGZA, seg_pos={0 0}, upload=174716]: Long tail cancellation due to optimal threshold Nov 7 17:55:02.658800281 (198h14m34.354649034s) [seg_pos={0 0}, upload=174716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:55:02.659857233 (198h14m34.355705986s) [upload=174716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:55:02.659867564 (198h14m34.355716317s) [upload=174716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:55:02.975488404 (198h14m34.671337157s) [upload=174716]: Done uploading segments... Nov 7 17:55:02.975541576 (198h14m34.671390329s) [upload=174716]: Upload finished. err: <nil> Duration: 30.007213151s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.567086179s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:54:52 UTC) orbiter warning: context deadline exceeded Duration: 3.567125023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..] duration: 23.586003164s trace: https://jaeger.dev.storj.io/trace/530d659d843959e9 FAIL (2025-11-07 17:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA) context deadline exceeded; piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA) context deadline exceeded ======LOGS====== Nov 7 17:52:12.562216144 (198h11m44.258064897s) [upload=174673]: Starting upload Nov 7 17:52:12.562328187 (198h11m44.258176941s) [upload=174673]: Done starting upload Nov 7 17:52:12.562469617 (198h11m44.25831836s) [upload=174673]: Uploading segments... Nov 7 17:52:12.562749781 (198h11m44.258598534s) [upload=174673]: Got next segment. Inline: false Nov 7 17:52:12.562804725 (198h11m44.258653478s) [upload=174673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:52:12.562823321 (198h11m44.258672074s) [upload=174673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:52:13.687331054 (198h11m45.383179847s) [seg_pos={0 0}, upload=174673]: Begin upload segment... Nov 7 17:52:13.687402631 (198h11m45.383251384s) [seg_pos={0 0}, upload=174673]: Done begin upload segment. Nov 7 17:52:13.687516888 (198h11m45.383365641s) [noise=true , address=45.91.116.6:4974 , piece_id=ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.687549189 (198h11m45.383397942s) [noise=true , address=77.90.31.19:4841 , piece_id=H4COH2MHLXIQPRBDVD4XYC6I6AWAEXWBIZOMV5VBELVGQTLTST5Q, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.687693866 (198h11m45.383542619s) [noise=true , address=145.79.227.10:4403 , piece_id=3A2VNIUN73KNVVX7DSTNPEXEHXIYLZKX4LQLOQBUZZTQG43ANVEQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.688468010 (198h11m45.384316763s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.688747592 (198h11m45.384596345s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.689195735 (198h11m45.385044488s) [noise=true , address=145.79.227.20:4438 , piece_id=PFDEFYXXORUK6GU67WWY7HGZISV32GXBKT5A7ANFWCT54YKF4ZJA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.689609413 (198h11m45.385458166s) [noise=true , address=77.90.31.21:4879 , piece_id=Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690277245 (198h11m45.386126008s) [noise=true , address=45.91.116.4:4940 , piece_id=FGWJEBHYZ3Q46HIZRSEJI3H6GZDWAIJ5B3ENEKNKPPN6EXX2C2TA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690317812 (198h11m45.386166565s) [noise=true , address=145.79.227.11:4732 , piece_id=566A7BABHOJPMXYOR3MB5H56YXPI4ITPVAVEB6TSO5HRQFGFK3UQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690747040 (198h11m45.386595853s) [noise=true , address=145.79.227.27:4690 , piece_id=INJWV7ZGYKXMSBJW3PKRAAQWQKJDFWXHW6KESAXGWACR6SHYG3WQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.691284924 (198h11m45.387133667s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7AJS4QMKQ77RPUID4DI5MINNAOCDRZM5NRGVKS7DYYJMHSNCVQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.692045061 (198h11m45.387893814s) [noise=true , address=145.79.227.30:4761 , piece_id=S3MVK6IKQFGZXXU7CCM56ZHTSCDTN7BTFMZ4FZXNECO2HXZLLQ6Q, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.692731929 (198h11m45.388580682s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.693792378 (198h11m45.389641141s) [noise=true , address=145.79.227.15:4874 , piece_id=N7MCIT566PUX5S7ZUPLFJ67M5QQ3FN2CKELIRKON5E5CCXNVUXNQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.694853529 (198h11m45.390702312s) [noise=true , address=77.90.31.15:4698 , piece_id=MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.695881907 (198h11m45.39173066s) [noise=true , address=145.79.227.32:4834 , piece_id=OTIPC3GO7Y5635GGQ27RHKCNOG73ZWQYBZ2T6CELINESMEFRWNZQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:15.361377676 (198h11m47.057226429s) [upload=174673]: Next returned nil segment Nov 7 17:52:15.361439905 (198h11m47.057288658s) [upload=174673]: Waiting for error group managing segments... Nov 7 17:52:15.636471035 (198h11m47.332319788s) [noise=true , address=145.79.227.27:4690 , piece_id=INJWV7ZGYKXMSBJW3PKRAAQWQKJDFWXHW6KESAXGWACR6SHYG3WQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.681418995 (198h11m47.377267768s) [noise=true , address=145.79.227.10:4403 , piece_id=3A2VNIUN73KNVVX7DSTNPEXEHXIYLZKX4LQLOQBUZZTQG43ANVEQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.868874291 (198h11m47.564723054s) [noise=true , address=145.79.227.15:4874 , piece_id=N7MCIT566PUX5S7ZUPLFJ67M5QQ3FN2CKELIRKON5E5CCXNVUXNQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.869668905 (198h11m47.565517648s) [noise=true , address=145.79.227.32:4834 , piece_id=OTIPC3GO7Y5635GGQ27RHKCNOG73ZWQYBZ2T6CELINESMEFRWNZQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.905313510 (198h11m47.601162253s) [noise=true , address=145.79.227.11:4732 , piece_id=566A7BABHOJPMXYOR3MB5H56YXPI4ITPVAVEB6TSO5HRQFGFK3UQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.963064941 (198h11m47.658913694s) [noise=true , address=145.79.227.30:4761 , piece_id=S3MVK6IKQFGZXXU7CCM56ZHTSCDTN7BTFMZ4FZXNECO2HXZLLQ6Q, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:16.013342881 (198h11m47.709191624s) [noise=true , address=145.79.227.20:4438 , piece_id=PFDEFYXXORUK6GU67WWY7HGZISV32GXBKT5A7ANFWCT54YKF4ZJA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:16.863684401 (198h11m48.559533164s) [noise=true , address=45.91.116.6:4974 , piece_id=ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.022366627 (198h11m48.71821537s) [noise=true , address=45.91.116.4:4940 , piece_id=FGWJEBHYZ3Q46HIZRSEJI3H6GZDWAIJ5B3ENEKNKPPN6EXX2C2TA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.230404297 (198h11m48.92625305s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7AJS4QMKQ77RPUID4DI5MINNAOCDRZM5NRGVKS7DYYJMHSNCVQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.660422869 (198h11m49.356271622s) [noise=true , address=77.90.31.21:4879 , piece_id=Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.148383151 (198h11m49.844231904s) [noise=true , address=77.90.31.15:4698 , piece_id=MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.298206969 (198h11m49.994055712s) [noise=true , address=77.90.31.19:4841 , piece_id=H4COH2MHLXIQPRBDVD4XYC6I6AWAEXWBIZOMV5VBELVGQTLTST5Q, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.298228730 (198h11m49.994077473s) [seg_pos={0 0}, upload=174673]: Segment reached optimal threshold of 13 pieces. Nov 7 17:52:18.298425034 (198h11m49.994273817s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298434642 (198h11m49.994283385s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.298698385 (198h11m49.994547138s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298708293 (198h11m49.994557046s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.298851536 (198h11m49.994700289s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298885911 (198h11m49.994734674s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.299096933 (198h11m49.994945686s) [seg_pos={0 0}, upload=174673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:52:18.299235617 (198h11m49.99508438s) [upload=174673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:52:18.299243652 (198h11m49.995092405s) [upload=174673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:52:18.823894524 (198h11m50.519743307s) [upload=174673]: Done uploading segments... Nov 7 17:52:18.823924952 (198h11m50.519773705s) [upload=174673]: Upload finished. err: <nil> Duration: 23.586024795s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.599099648s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:54:00 UTC) orbiter warning: context deadline exceeded Duration: 3.599130898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X.] duration: 19.848119591s trace: https://jaeger.dev.storj.io/trace/530d659d843959e9 FAIL (2025-11-07 17:53:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA) context deadline exceeded; piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA) context deadline exceeded ======LOGS====== Nov 7 17:52:12.562216144 (198h11m44.258064897s) [upload=174673]: Starting upload Nov 7 17:52:12.562328187 (198h11m44.258176941s) [upload=174673]: Done starting upload Nov 7 17:52:12.562469617 (198h11m44.25831836s) [upload=174673]: Uploading segments... Nov 7 17:52:12.562749781 (198h11m44.258598534s) [upload=174673]: Got next segment. Inline: false Nov 7 17:52:12.562804725 (198h11m44.258653478s) [upload=174673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:52:12.562823321 (198h11m44.258672074s) [upload=174673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:52:13.687331054 (198h11m45.383179847s) [seg_pos={0 0}, upload=174673]: Begin upload segment... Nov 7 17:52:13.687402631 (198h11m45.383251384s) [seg_pos={0 0}, upload=174673]: Done begin upload segment. Nov 7 17:52:13.687516888 (198h11m45.383365641s) [noise=true , address=45.91.116.6:4974 , piece_id=ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.687549189 (198h11m45.383397942s) [noise=true , address=77.90.31.19:4841 , piece_id=H4COH2MHLXIQPRBDVD4XYC6I6AWAEXWBIZOMV5VBELVGQTLTST5Q, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.687693866 (198h11m45.383542619s) [noise=true , address=145.79.227.10:4403 , piece_id=3A2VNIUN73KNVVX7DSTNPEXEHXIYLZKX4LQLOQBUZZTQG43ANVEQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.688468010 (198h11m45.384316763s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.688747592 (198h11m45.384596345s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.689195735 (198h11m45.385044488s) [noise=true , address=145.79.227.20:4438 , piece_id=PFDEFYXXORUK6GU67WWY7HGZISV32GXBKT5A7ANFWCT54YKF4ZJA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.689609413 (198h11m45.385458166s) [noise=true , address=77.90.31.21:4879 , piece_id=Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690277245 (198h11m45.386126008s) [noise=true , address=45.91.116.4:4940 , piece_id=FGWJEBHYZ3Q46HIZRSEJI3H6GZDWAIJ5B3ENEKNKPPN6EXX2C2TA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690317812 (198h11m45.386166565s) [noise=true , address=145.79.227.11:4732 , piece_id=566A7BABHOJPMXYOR3MB5H56YXPI4ITPVAVEB6TSO5HRQFGFK3UQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690747040 (198h11m45.386595853s) [noise=true , address=145.79.227.27:4690 , piece_id=INJWV7ZGYKXMSBJW3PKRAAQWQKJDFWXHW6KESAXGWACR6SHYG3WQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.691284924 (198h11m45.387133667s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7AJS4QMKQ77RPUID4DI5MINNAOCDRZM5NRGVKS7DYYJMHSNCVQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.692045061 (198h11m45.387893814s) [noise=true , address=145.79.227.30:4761 , piece_id=S3MVK6IKQFGZXXU7CCM56ZHTSCDTN7BTFMZ4FZXNECO2HXZLLQ6Q, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.692731929 (198h11m45.388580682s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.693792378 (198h11m45.389641141s) [noise=true , address=145.79.227.15:4874 , piece_id=N7MCIT566PUX5S7ZUPLFJ67M5QQ3FN2CKELIRKON5E5CCXNVUXNQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.694853529 (198h11m45.390702312s) [noise=true , address=77.90.31.15:4698 , piece_id=MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.695881907 (198h11m45.39173066s) [noise=true , address=145.79.227.32:4834 , piece_id=OTIPC3GO7Y5635GGQ27RHKCNOG73ZWQYBZ2T6CELINESMEFRWNZQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:15.361377676 (198h11m47.057226429s) [upload=174673]: Next returned nil segment Nov 7 17:52:15.361439905 (198h11m47.057288658s) [upload=174673]: Waiting for error group managing segments... Nov 7 17:52:15.636471035 (198h11m47.332319788s) [noise=true , address=145.79.227.27:4690 , piece_id=INJWV7ZGYKXMSBJW3PKRAAQWQKJDFWXHW6KESAXGWACR6SHYG3WQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.681418995 (198h11m47.377267768s) [noise=true , address=145.79.227.10:4403 , piece_id=3A2VNIUN73KNVVX7DSTNPEXEHXIYLZKX4LQLOQBUZZTQG43ANVEQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.868874291 (198h11m47.564723054s) [noise=true , address=145.79.227.15:4874 , piece_id=N7MCIT566PUX5S7ZUPLFJ67M5QQ3FN2CKELIRKON5E5CCXNVUXNQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.869668905 (198h11m47.565517648s) [noise=true , address=145.79.227.32:4834 , piece_id=OTIPC3GO7Y5635GGQ27RHKCNOG73ZWQYBZ2T6CELINESMEFRWNZQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.905313510 (198h11m47.601162253s) [noise=true , address=145.79.227.11:4732 , piece_id=566A7BABHOJPMXYOR3MB5H56YXPI4ITPVAVEB6TSO5HRQFGFK3UQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.963064941 (198h11m47.658913694s) [noise=true , address=145.79.227.30:4761 , piece_id=S3MVK6IKQFGZXXU7CCM56ZHTSCDTN7BTFMZ4FZXNECO2HXZLLQ6Q, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:16.013342881 (198h11m47.709191624s) [noise=true , address=145.79.227.20:4438 , piece_id=PFDEFYXXORUK6GU67WWY7HGZISV32GXBKT5A7ANFWCT54YKF4ZJA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:16.863684401 (198h11m48.559533164s) [noise=true , address=45.91.116.6:4974 , piece_id=ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.022366627 (198h11m48.71821537s) [noise=true , address=45.91.116.4:4940 , piece_id=FGWJEBHYZ3Q46HIZRSEJI3H6GZDWAIJ5B3ENEKNKPPN6EXX2C2TA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.230404297 (198h11m48.92625305s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7AJS4QMKQ77RPUID4DI5MINNAOCDRZM5NRGVKS7DYYJMHSNCVQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.660422869 (198h11m49.356271622s) [noise=true , address=77.90.31.21:4879 , piece_id=Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.148383151 (198h11m49.844231904s) [noise=true , address=77.90.31.15:4698 , piece_id=MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.298206969 (198h11m49.994055712s) [noise=true , address=77.90.31.19:4841 , piece_id=H4COH2MHLXIQPRBDVD4XYC6I6AWAEXWBIZOMV5VBELVGQTLTST5Q, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.298228730 (198h11m49.994077473s) [seg_pos={0 0}, upload=174673]: Segment reached optimal threshold of 13 pieces. Nov 7 17:52:18.298425034 (198h11m49.994273817s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298434642 (198h11m49.994283385s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.298698385 (198h11m49.994547138s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298708293 (198h11m49.994557046s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.298851536 (198h11m49.994700289s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298885911 (198h11m49.994734674s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.299096933 (198h11m49.994945686s) [seg_pos={0 0}, upload=174673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:52:18.299235617 (198h11m49.99508438s) [upload=174673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:52:18.299243652 (198h11m49.995092405s) [upload=174673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:52:18.823894524 (198h11m50.519743307s) [upload=174673]: Done uploading segments... Nov 7 17:52:18.823924952 (198h11m50.519773705s) [upload=174673]: Upload finished. err: <nil> Duration: 19.84816117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.353173369s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:53:11 UTC) orbiter warning: context deadline exceeded Duration: 3.353616483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.485577624s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:52:45 UTC) orbiter warning: context deadline exceeded Duration: 3.485602774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.....................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X] duration: 30.004114556s trace: https://jaeger.dev.storj.io/trace/530d659d843959e9 FAIL (2025-11-07 17:52:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA) context deadline exceeded; piecestore: (Node ID: 12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, Piece ID: Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA) context deadline exceeded ======LOGS====== Nov 7 17:52:12.562216144 (198h11m44.258064897s) [upload=174673]: Starting upload Nov 7 17:52:12.562328187 (198h11m44.258176941s) [upload=174673]: Done starting upload Nov 7 17:52:12.562469617 (198h11m44.25831836s) [upload=174673]: Uploading segments... Nov 7 17:52:12.562749781 (198h11m44.258598534s) [upload=174673]: Got next segment. Inline: false Nov 7 17:52:12.562804725 (198h11m44.258653478s) [upload=174673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:52:12.562823321 (198h11m44.258672074s) [upload=174673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:52:13.687331054 (198h11m45.383179847s) [seg_pos={0 0}, upload=174673]: Begin upload segment... Nov 7 17:52:13.687402631 (198h11m45.383251384s) [seg_pos={0 0}, upload=174673]: Done begin upload segment. Nov 7 17:52:13.687516888 (198h11m45.383365641s) [noise=true , address=45.91.116.6:4974 , piece_id=ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.687549189 (198h11m45.383397942s) [noise=true , address=77.90.31.19:4841 , piece_id=H4COH2MHLXIQPRBDVD4XYC6I6AWAEXWBIZOMV5VBELVGQTLTST5Q, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.687693866 (198h11m45.383542619s) [noise=true , address=145.79.227.10:4403 , piece_id=3A2VNIUN73KNVVX7DSTNPEXEHXIYLZKX4LQLOQBUZZTQG43ANVEQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.688468010 (198h11m45.384316763s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.688747592 (198h11m45.384596345s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.689195735 (198h11m45.385044488s) [noise=true , address=145.79.227.20:4438 , piece_id=PFDEFYXXORUK6GU67WWY7HGZISV32GXBKT5A7ANFWCT54YKF4ZJA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.689609413 (198h11m45.385458166s) [noise=true , address=77.90.31.21:4879 , piece_id=Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690277245 (198h11m45.386126008s) [noise=true , address=45.91.116.4:4940 , piece_id=FGWJEBHYZ3Q46HIZRSEJI3H6GZDWAIJ5B3ENEKNKPPN6EXX2C2TA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690317812 (198h11m45.386166565s) [noise=true , address=145.79.227.11:4732 , piece_id=566A7BABHOJPMXYOR3MB5H56YXPI4ITPVAVEB6TSO5HRQFGFK3UQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.690747040 (198h11m45.386595853s) [noise=true , address=145.79.227.27:4690 , piece_id=INJWV7ZGYKXMSBJW3PKRAAQWQKJDFWXHW6KESAXGWACR6SHYG3WQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.691284924 (198h11m45.387133667s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7AJS4QMKQ77RPUID4DI5MINNAOCDRZM5NRGVKS7DYYJMHSNCVQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.692045061 (198h11m45.387893814s) [noise=true , address=145.79.227.30:4761 , piece_id=S3MVK6IKQFGZXXU7CCM56ZHTSCDTN7BTFMZ4FZXNECO2HXZLLQ6Q, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.692731929 (198h11m45.388580682s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.693792378 (198h11m45.389641141s) [noise=true , address=145.79.227.15:4874 , piece_id=N7MCIT566PUX5S7ZUPLFJ67M5QQ3FN2CKELIRKON5E5CCXNVUXNQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.694853529 (198h11m45.390702312s) [noise=true , address=77.90.31.15:4698 , piece_id=MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:13.695881907 (198h11m45.39173066s) [noise=true , address=145.79.227.32:4834 , piece_id=OTIPC3GO7Y5635GGQ27RHKCNOG73ZWQYBZ2T6CELINESMEFRWNZQ, seg_pos={0 0}, upload=174673]: Uploading piece... Nov 7 17:52:15.361377676 (198h11m47.057226429s) [upload=174673]: Next returned nil segment Nov 7 17:52:15.361439905 (198h11m47.057288658s) [upload=174673]: Waiting for error group managing segments... Nov 7 17:52:15.636471035 (198h11m47.332319788s) [noise=true , address=145.79.227.27:4690 , piece_id=INJWV7ZGYKXMSBJW3PKRAAQWQKJDFWXHW6KESAXGWACR6SHYG3WQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.681418995 (198h11m47.377267768s) [noise=true , address=145.79.227.10:4403 , piece_id=3A2VNIUN73KNVVX7DSTNPEXEHXIYLZKX4LQLOQBUZZTQG43ANVEQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.868874291 (198h11m47.564723054s) [noise=true , address=145.79.227.15:4874 , piece_id=N7MCIT566PUX5S7ZUPLFJ67M5QQ3FN2CKELIRKON5E5CCXNVUXNQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.869668905 (198h11m47.565517648s) [noise=true , address=145.79.227.32:4834 , piece_id=OTIPC3GO7Y5635GGQ27RHKCNOG73ZWQYBZ2T6CELINESMEFRWNZQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.905313510 (198h11m47.601162253s) [noise=true , address=145.79.227.11:4732 , piece_id=566A7BABHOJPMXYOR3MB5H56YXPI4ITPVAVEB6TSO5HRQFGFK3UQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:15.963064941 (198h11m47.658913694s) [noise=true , address=145.79.227.30:4761 , piece_id=S3MVK6IKQFGZXXU7CCM56ZHTSCDTN7BTFMZ4FZXNECO2HXZLLQ6Q, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:16.013342881 (198h11m47.709191624s) [noise=true , address=145.79.227.20:4438 , piece_id=PFDEFYXXORUK6GU67WWY7HGZISV32GXBKT5A7ANFWCT54YKF4ZJA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:16.863684401 (198h11m48.559533164s) [noise=true , address=45.91.116.6:4974 , piece_id=ZJ3EOLVHYQH2NOGSLDVFVPYAH72Z5LHVWNMP7PXTUIQNUSBBUMXQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.022366627 (198h11m48.71821537s) [noise=true , address=45.91.116.4:4940 , piece_id=FGWJEBHYZ3Q46HIZRSEJI3H6GZDWAIJ5B3ENEKNKPPN6EXX2C2TA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.230404297 (198h11m48.92625305s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7AJS4QMKQ77RPUID4DI5MINNAOCDRZM5NRGVKS7DYYJMHSNCVQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:17.660422869 (198h11m49.356271622s) [noise=true , address=77.90.31.21:4879 , piece_id=Q7ECRCX366TUTQONY2TPHDZ4ACPLZ32EYZ24CRB6YK5PA7UWWPAA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.148383151 (198h11m49.844231904s) [noise=true , address=77.90.31.15:4698 , piece_id=MCOSJTB4VGB4ARQ7DJI45TYWEQOHSQQNHJYKKKDZLLTW7KAGQWIA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.298206969 (198h11m49.994055712s) [noise=true , address=77.90.31.19:4841 , piece_id=H4COH2MHLXIQPRBDVD4XYC6I6AWAEXWBIZOMV5VBELVGQTLTST5Q, seg_pos={0 0}, upload=174673]: Done uploading piece. err: <nil> Nov 7 17:52:18.298228730 (198h11m49.994077473s) [seg_pos={0 0}, upload=174673]: Segment reached optimal threshold of 13 pieces. Nov 7 17:52:18.298425034 (198h11m49.994273817s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298434642 (198h11m49.994283385s) [noise=true , address=45.91.116.10:4083 , piece_id=F6TS6TFEASODH44TVHCFZ7QFHNLL3OKI6VQCGD3N7R25QF5ON5FA, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.298698385 (198h11m49.994547138s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298708293 (198h11m49.994557046s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYPRLUWSUQR65THSFCFHA3Q4MVWEEPDJO5L32AYZHYIUG2U7LRNQ, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.298851536 (198h11m49.994700289s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Nov 7 17:52:18.298885911 (198h11m49.994734674s) [noise=true , address=77.90.31.13:4626 , piece_id=G5ZMKG4C2444MAMDCSTJCTPT4YHAVEL7XDCXUGF3HMDLY2AEVYKQ, seg_pos={0 0}, upload=174673]: Long tail cancellation due to optimal threshold Nov 7 17:52:18.299096933 (198h11m49.994945686s) [seg_pos={0 0}, upload=174673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:52:18.299235617 (198h11m49.99508438s) [upload=174673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:52:18.299243652 (198h11m49.995092405s) [upload=174673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:52:18.823894524 (198h11m50.519743307s) [upload=174673]: Done uploading segments... Nov 7 17:52:18.823924952 (198h11m50.519773705s) [upload=174673]: Upload finished. err: <nil> Duration: 30.004162717s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.495196882s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:52:01 UTC) orbiter warning: context deadline exceeded Duration: 3.495246496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [......................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.] duration: 15.300581496s trace: https://jaeger.dev.storj.io/trace/138a5e4ede9ff079 FAIL (2025-11-07 17:51:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: GKA3JAW3IN7OFKVXDN6MBOBKLB7PINGHBBTDI6SB2KPWXUWBENJA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: L5NN5MUWEPF7JHBM7PK4Y76RDLJ2TXG6PHUYB4XB6CTQDMZF5XQQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HDIQYXNGVMK5BSVISSMBUZRRPA6B6PQNMM3FOSIKM5PQVZUY7FYA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3LUWG7J5LYMQZT7B6ULCFZZMYRUS6E5E6RG5XBV2AXUNGAULN2PA) context deadline exceeded ======LOGS====== Nov 7 17:50:50.022105508 (198h10m21.717954251s) [upload=174653]: Starting upload Nov 7 17:50:50.022202884 (198h10m21.718051627s) [upload=174653]: Done starting upload Nov 7 17:50:50.022246316 (198h10m21.718095069s) [upload=174653]: Uploading segments... Nov 7 17:50:50.022558270 (198h10m21.718407023s) [upload=174653]: Got next segment. Inline: false Nov 7 17:50:50.022611422 (198h10m21.718460175s) [upload=174653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:50:50.022618736 (198h10m21.718467489s) [upload=174653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:50:53.047135864 (198h10m24.742984627s) [seg_pos={0 0}, upload=174653]: Begin upload segment... Nov 7 17:50:53.047210255 (198h10m24.743059008s) [seg_pos={0 0}, upload=174653]: Done begin upload segment. Nov 7 17:50:53.047259389 (198h10m24.743108142s) [noise=true , address=145.79.227.11:4730 , piece_id=KHXTTUOOMDBGGYJ5TEIZM4TRPPUFTGNZWD3MOW2IBHYNENOV6OZQ, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.047318212 (198h10m24.743166995s) [noise=true , address=45.91.116.4:4939 , piece_id=FSSPCKKYB5MJHERQ2SQKESEDI5RZCJUXFZIRFOPTCVLTVEO253YA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.047882626 (198h10m24.743731379s) [noise=true , address=45.91.116.10:4082 , piece_id=HDIQYXNGVMK5BSVISSMBUZRRPA6B6PQNMM3FOSIKM5PQVZUY7FYA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.047892074 (198h10m24.743740827s) [noise=true , address=45.91.116.14:4190 , piece_id=3LUWG7J5LYMQZT7B6ULCFZZMYRUS6E5E6RG5XBV2AXUNGAULN2PA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.048366528 (198h10m24.744215271s) [noise=true , address=145.79.227.23:4545 , piece_id=XGOQU6TAHCQQLABEBWHCCLGE5WCPEGBD7PYRRR5JSRZDTAGTJX4A, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.048384371 (198h10m24.744233124s) [noise=true , address=145.79.227.32:4836 , piece_id=CUQIQTVRO6OZXQCMEVJDYDT5JMARLK6UNWZ45VVO3ZVXAVA7SQKA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.049572403 (198h10m24.745421156s) [noise=true , address=145.79.227.10:4402 , piece_id=GPB6LN6HJZQMMPRHF4JXZ5NLT2H5BW4FXKUP2SECRUETXGQ6ENHA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.049672354 (198h10m24.745521107s) [noise=true , address=145.79.227.8:4331 , piece_id=4QPCF6MZJUYA3YJNUVTEHGC7FRFESMBH4HPFIYM5BL34MMKCTEKA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.051083932 (198h10m24.746932685s) [noise=true , address=77.90.31.12:4590 , piece_id=JMGHQ7GMEEQLHDNOPW5J6MPX2MZVVOUHXJUMH7SIZONGODDZMVHA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.051482420 (198h10m24.747331173s) [noise=true , address=45.91.116.5:4299 , piece_id=FZD2VHYS4PKJZC5OYPKKIGMZUFC64M4UG7E57VG6QMNRK6RS3J4Q, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.052567276 (198h10m24.748416029s) [noise=true , address=77.90.31.18:4805 , piece_id=OY2GDLXLGFHNF7QJBA5FFK2SFJQ34NTQZ64PJX5CQM6XSSMAL4XA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.053220129 (198h10m24.749068882s) [noise=true , address=77.90.31.19:4841 , piece_id=GKA3JAW3IN7OFKVXDN6MBOBKLB7PINGHBBTDI6SB2KPWXUWBENJA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.054248167 (198h10m24.75009692s) [noise=true , address=145.79.227.9:4365 , piece_id=OIZ3EY364Q6LQIVM3R2VE6BZJCA34KVFYON7MHLDF3DWSBGFRQXA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.054856716 (198h10m24.750705489s) [noise=true , address=77.90.31.11:4554 , piece_id=L5NN5MUWEPF7JHBM7PK4Y76RDLJ2TXG6PHUYB4XB6CTQDMZF5XQQ, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.055303657 (198h10m24.75115241s) [noise=true , address=77.90.31.21:4879 , piece_id=BNH7W2JWL5SXQ26433YO2VMKERWYYQDZIOFW7ANJTY4K6ZA3OS3A, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.055995725 (198h10m24.751844488s) [noise=true , address=145.79.227.30:4762 , piece_id=QHYMJDSCWIAXOYGLGH3YYT56YJZLBZ4LUDOJB7RPKFINVRAU5SFA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:54.892236341 (198h10m26.588085084s) [upload=174653]: Next returned nil segment Nov 7 17:50:54.892256780 (198h10m26.588105543s) [upload=174653]: Waiting for error group managing segments... Nov 7 17:50:55.173508042 (198h10m26.869356795s) [noise=true , address=145.79.227.11:4730 , piece_id=KHXTTUOOMDBGGYJ5TEIZM4TRPPUFTGNZWD3MOW2IBHYNENOV6OZQ, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.245571016 (198h10m26.941419779s) [noise=true , address=145.79.227.32:4836 , piece_id=CUQIQTVRO6OZXQCMEVJDYDT5JMARLK6UNWZ45VVO3ZVXAVA7SQKA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.251612330 (198h10m26.947461083s) [noise=true , address=145.79.227.8:4331 , piece_id=4QPCF6MZJUYA3YJNUVTEHGC7FRFESMBH4HPFIYM5BL34MMKCTEKA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.258056180 (198h10m26.953904923s) [noise=true , address=145.79.227.9:4365 , piece_id=OIZ3EY364Q6LQIVM3R2VE6BZJCA34KVFYON7MHLDF3DWSBGFRQXA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.276419297 (198h10m26.97226805s) [noise=true , address=45.91.116.4:4939 , piece_id=FSSPCKKYB5MJHERQ2SQKESEDI5RZCJUXFZIRFOPTCVLTVEO253YA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.325677195 (198h10m27.021525948s) [noise=true , address=45.91.116.10:4082 , piece_id=HDIQYXNGVMK5BSVISSMBUZRRPA6B6PQNMM3FOSIKM5PQVZUY7FYA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.333837103 (198h10m27.029685846s) [noise=true , address=45.91.116.5:4299 , piece_id=FZD2VHYS4PKJZC5OYPKKIGMZUFC64M4UG7E57VG6QMNRK6RS3J4Q, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.419654628 (198h10m27.115503381s) [noise=true , address=145.79.227.30:4762 , piece_id=QHYMJDSCWIAXOYGLGH3YYT56YJZLBZ4LUDOJB7RPKFINVRAU5SFA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.438766450 (198h10m27.134615193s) [noise=true , address=77.90.31.11:4554 , piece_id=L5NN5MUWEPF7JHBM7PK4Y76RDLJ2TXG6PHUYB4XB6CTQDMZF5XQQ, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.510843661 (198h10m27.206692414s) [noise=true , address=45.91.116.14:4190 , piece_id=3LUWG7J5LYMQZT7B6ULCFZZMYRUS6E5E6RG5XBV2AXUNGAULN2PA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.778166962 (198h10m27.474015715s) [noise=true , address=145.79.227.23:4545 , piece_id=XGOQU6TAHCQQLABEBWHCCLGE5WCPEGBD7PYRRR5JSRZDTAGTJX4A, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.819272846 (198h10m27.515121599s) [noise=true , address=145.79.227.10:4402 , piece_id=GPB6LN6HJZQMMPRHF4JXZ5NLT2H5BW4FXKUP2SECRUETXGQ6ENHA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:56.008553999 (198h10m27.704402752s) [noise=true , address=77.90.31.19:4841 , piece_id=GKA3JAW3IN7OFKVXDN6MBOBKLB7PINGHBBTDI6SB2KPWXUWBENJA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:56.008582493 (198h10m27.704431246s) [seg_pos={0 0}, upload=174653]: Segment reached optimal threshold of 13 pieces. Nov 7 17:50:56.008826608 (198h10m27.704675361s) [noise=true , address=77.90.31.18:4805 , piece_id=OY2GDLXLGFHNF7QJBA5FFK2SFJQ34NTQZ64PJX5CQM6XSSMAL4XA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:56.008835004 (198h10m27.704683747s) [noise=true , address=77.90.31.18:4805 , piece_id=OY2GDLXLGFHNF7QJBA5FFK2SFJQ34NTQZ64PJX5CQM6XSSMAL4XA, seg_pos={0 0}, upload=174653]: Long tail cancellation due to optimal threshold Nov 7 17:50:56.009065132 (198h10m27.704913895s) [noise=true , address=77.90.31.12:4590 , piece_id=JMGHQ7GMEEQLHDNOPW5J6MPX2MZVVOUHXJUMH7SIZONGODDZMVHA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:56.009087244 (198h10m27.704935997s) [noise=true , address=77.90.31.12:4590 , piece_id=JMGHQ7GMEEQLHDNOPW5J6MPX2MZVVOUHXJUMH7SIZONGODDZMVHA, seg_pos={0 0}, upload=174653]: Long tail cancellation due to optimal threshold Nov 7 17:50:56.009129885 (198h10m27.704978648s) [noise=true , address=77.90.31.21:4879 , piece_id=BNH7W2JWL5SXQ26433YO2VMKERWYYQDZIOFW7ANJTY4K6ZA3OS3A, seg_pos={0 0}, upload=174653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:56.009139324 (198h10m27.704988077s) [noise=true , address=77.90.31.21:4879 , piece_id=BNH7W2JWL5SXQ26433YO2VMKERWYYQDZIOFW7ANJTY4K6ZA3OS3A, seg_pos={0 0}, upload=174653]: Long tail cancellation due to optimal threshold Nov 7 17:50:56.009194648 (198h10m27.705043401s) [seg_pos={0 0}, upload=174653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:50:56.009290701 (198h10m27.705139454s) [upload=174653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:50:56.009297644 (198h10m27.705146398s) [upload=174653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:50:56.652501462 (198h10m28.348350225s) [upload=174653]: Done uploading segments... Nov 7 17:50:56.652525368 (198h10m28.348374121s) [upload=174653]: Upload finished. err: <nil> Duration: 15.300616864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.609211451s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:51:40 UTC) orbiter warning: context deadline exceeded Duration: 3.609240315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.......................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX] duration: 30.002974005s trace: https://jaeger.dev.storj.io/trace/138a5e4ede9ff079 FAIL (2025-11-07 17:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: GKA3JAW3IN7OFKVXDN6MBOBKLB7PINGHBBTDI6SB2KPWXUWBENJA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: L5NN5MUWEPF7JHBM7PK4Y76RDLJ2TXG6PHUYB4XB6CTQDMZF5XQQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: HDIQYXNGVMK5BSVISSMBUZRRPA6B6PQNMM3FOSIKM5PQVZUY7FYA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3LUWG7J5LYMQZT7B6ULCFZZMYRUS6E5E6RG5XBV2AXUNGAULN2PA) context deadline exceeded ======LOGS====== Nov 7 17:50:50.022105508 (198h10m21.717954251s) [upload=174653]: Starting upload Nov 7 17:50:50.022202884 (198h10m21.718051627s) [upload=174653]: Done starting upload Nov 7 17:50:50.022246316 (198h10m21.718095069s) [upload=174653]: Uploading segments... Nov 7 17:50:50.022558270 (198h10m21.718407023s) [upload=174653]: Got next segment. Inline: false Nov 7 17:50:50.022611422 (198h10m21.718460175s) [upload=174653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:50:50.022618736 (198h10m21.718467489s) [upload=174653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:50:53.047135864 (198h10m24.742984627s) [seg_pos={0 0}, upload=174653]: Begin upload segment... Nov 7 17:50:53.047210255 (198h10m24.743059008s) [seg_pos={0 0}, upload=174653]: Done begin upload segment. Nov 7 17:50:53.047259389 (198h10m24.743108142s) [noise=true , address=145.79.227.11:4730 , piece_id=KHXTTUOOMDBGGYJ5TEIZM4TRPPUFTGNZWD3MOW2IBHYNENOV6OZQ, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.047318212 (198h10m24.743166995s) [noise=true , address=45.91.116.4:4939 , piece_id=FSSPCKKYB5MJHERQ2SQKESEDI5RZCJUXFZIRFOPTCVLTVEO253YA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.047882626 (198h10m24.743731379s) [noise=true , address=45.91.116.10:4082 , piece_id=HDIQYXNGVMK5BSVISSMBUZRRPA6B6PQNMM3FOSIKM5PQVZUY7FYA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.047892074 (198h10m24.743740827s) [noise=true , address=45.91.116.14:4190 , piece_id=3LUWG7J5LYMQZT7B6ULCFZZMYRUS6E5E6RG5XBV2AXUNGAULN2PA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.048366528 (198h10m24.744215271s) [noise=true , address=145.79.227.23:4545 , piece_id=XGOQU6TAHCQQLABEBWHCCLGE5WCPEGBD7PYRRR5JSRZDTAGTJX4A, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.048384371 (198h10m24.744233124s) [noise=true , address=145.79.227.32:4836 , piece_id=CUQIQTVRO6OZXQCMEVJDYDT5JMARLK6UNWZ45VVO3ZVXAVA7SQKA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.049572403 (198h10m24.745421156s) [noise=true , address=145.79.227.10:4402 , piece_id=GPB6LN6HJZQMMPRHF4JXZ5NLT2H5BW4FXKUP2SECRUETXGQ6ENHA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.049672354 (198h10m24.745521107s) [noise=true , address=145.79.227.8:4331 , piece_id=4QPCF6MZJUYA3YJNUVTEHGC7FRFESMBH4HPFIYM5BL34MMKCTEKA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.051083932 (198h10m24.746932685s) [noise=true , address=77.90.31.12:4590 , piece_id=JMGHQ7GMEEQLHDNOPW5J6MPX2MZVVOUHXJUMH7SIZONGODDZMVHA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.051482420 (198h10m24.747331173s) [noise=true , address=45.91.116.5:4299 , piece_id=FZD2VHYS4PKJZC5OYPKKIGMZUFC64M4UG7E57VG6QMNRK6RS3J4Q, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.052567276 (198h10m24.748416029s) [noise=true , address=77.90.31.18:4805 , piece_id=OY2GDLXLGFHNF7QJBA5FFK2SFJQ34NTQZ64PJX5CQM6XSSMAL4XA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.053220129 (198h10m24.749068882s) [noise=true , address=77.90.31.19:4841 , piece_id=GKA3JAW3IN7OFKVXDN6MBOBKLB7PINGHBBTDI6SB2KPWXUWBENJA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.054248167 (198h10m24.75009692s) [noise=true , address=145.79.227.9:4365 , piece_id=OIZ3EY364Q6LQIVM3R2VE6BZJCA34KVFYON7MHLDF3DWSBGFRQXA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.054856716 (198h10m24.750705489s) [noise=true , address=77.90.31.11:4554 , piece_id=L5NN5MUWEPF7JHBM7PK4Y76RDLJ2TXG6PHUYB4XB6CTQDMZF5XQQ, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.055303657 (198h10m24.75115241s) [noise=true , address=77.90.31.21:4879 , piece_id=BNH7W2JWL5SXQ26433YO2VMKERWYYQDZIOFW7ANJTY4K6ZA3OS3A, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:53.055995725 (198h10m24.751844488s) [noise=true , address=145.79.227.30:4762 , piece_id=QHYMJDSCWIAXOYGLGH3YYT56YJZLBZ4LUDOJB7RPKFINVRAU5SFA, seg_pos={0 0}, upload=174653]: Uploading piece... Nov 7 17:50:54.892236341 (198h10m26.588085084s) [upload=174653]: Next returned nil segment Nov 7 17:50:54.892256780 (198h10m26.588105543s) [upload=174653]: Waiting for error group managing segments... Nov 7 17:50:55.173508042 (198h10m26.869356795s) [noise=true , address=145.79.227.11:4730 , piece_id=KHXTTUOOMDBGGYJ5TEIZM4TRPPUFTGNZWD3MOW2IBHYNENOV6OZQ, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.245571016 (198h10m26.941419779s) [noise=true , address=145.79.227.32:4836 , piece_id=CUQIQTVRO6OZXQCMEVJDYDT5JMARLK6UNWZ45VVO3ZVXAVA7SQKA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.251612330 (198h10m26.947461083s) [noise=true , address=145.79.227.8:4331 , piece_id=4QPCF6MZJUYA3YJNUVTEHGC7FRFESMBH4HPFIYM5BL34MMKCTEKA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.258056180 (198h10m26.953904923s) [noise=true , address=145.79.227.9:4365 , piece_id=OIZ3EY364Q6LQIVM3R2VE6BZJCA34KVFYON7MHLDF3DWSBGFRQXA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.276419297 (198h10m26.97226805s) [noise=true , address=45.91.116.4:4939 , piece_id=FSSPCKKYB5MJHERQ2SQKESEDI5RZCJUXFZIRFOPTCVLTVEO253YA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.325677195 (198h10m27.021525948s) [noise=true , address=45.91.116.10:4082 , piece_id=HDIQYXNGVMK5BSVISSMBUZRRPA6B6PQNMM3FOSIKM5PQVZUY7FYA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.333837103 (198h10m27.029685846s) [noise=true , address=45.91.116.5:4299 , piece_id=FZD2VHYS4PKJZC5OYPKKIGMZUFC64M4UG7E57VG6QMNRK6RS3J4Q, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.419654628 (198h10m27.115503381s) [noise=true , address=145.79.227.30:4762 , piece_id=QHYMJDSCWIAXOYGLGH3YYT56YJZLBZ4LUDOJB7RPKFINVRAU5SFA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.438766450 (198h10m27.134615193s) [noise=true , address=77.90.31.11:4554 , piece_id=L5NN5MUWEPF7JHBM7PK4Y76RDLJ2TXG6PHUYB4XB6CTQDMZF5XQQ, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.510843661 (198h10m27.206692414s) [noise=true , address=45.91.116.14:4190 , piece_id=3LUWG7J5LYMQZT7B6ULCFZZMYRUS6E5E6RG5XBV2AXUNGAULN2PA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.778166962 (198h10m27.474015715s) [noise=true , address=145.79.227.23:4545 , piece_id=XGOQU6TAHCQQLABEBWHCCLGE5WCPEGBD7PYRRR5JSRZDTAGTJX4A, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:55.819272846 (198h10m27.515121599s) [noise=true , address=145.79.227.10:4402 , piece_id=GPB6LN6HJZQMMPRHF4JXZ5NLT2H5BW4FXKUP2SECRUETXGQ6ENHA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:56.008553999 (198h10m27.704402752s) [noise=true , address=77.90.31.19:4841 , piece_id=GKA3JAW3IN7OFKVXDN6MBOBKLB7PINGHBBTDI6SB2KPWXUWBENJA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: <nil> Nov 7 17:50:56.008582493 (198h10m27.704431246s) [seg_pos={0 0}, upload=174653]: Segment reached optimal threshold of 13 pieces. Nov 7 17:50:56.008826608 (198h10m27.704675361s) [noise=true , address=77.90.31.18:4805 , piece_id=OY2GDLXLGFHNF7QJBA5FFK2SFJQ34NTQZ64PJX5CQM6XSSMAL4XA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:56.008835004 (198h10m27.704683747s) [noise=true , address=77.90.31.18:4805 , piece_id=OY2GDLXLGFHNF7QJBA5FFK2SFJQ34NTQZ64PJX5CQM6XSSMAL4XA, seg_pos={0 0}, upload=174653]: Long tail cancellation due to optimal threshold Nov 7 17:50:56.009065132 (198h10m27.704913895s) [noise=true , address=77.90.31.12:4590 , piece_id=JMGHQ7GMEEQLHDNOPW5J6MPX2MZVVOUHXJUMH7SIZONGODDZMVHA, seg_pos={0 0}, upload=174653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:56.009087244 (198h10m27.704935997s) [noise=true , address=77.90.31.12:4590 , piece_id=JMGHQ7GMEEQLHDNOPW5J6MPX2MZVVOUHXJUMH7SIZONGODDZMVHA, seg_pos={0 0}, upload=174653]: Long tail cancellation due to optimal threshold Nov 7 17:50:56.009129885 (198h10m27.704978648s) [noise=true , address=77.90.31.21:4879 , piece_id=BNH7W2JWL5SXQ26433YO2VMKERWYYQDZIOFW7ANJTY4K6ZA3OS3A, seg_pos={0 0}, upload=174653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:56.009139324 (198h10m27.704988077s) [noise=true , address=77.90.31.21:4879 , piece_id=BNH7W2JWL5SXQ26433YO2VMKERWYYQDZIOFW7ANJTY4K6ZA3OS3A, seg_pos={0 0}, upload=174653]: Long tail cancellation due to optimal threshold Nov 7 17:50:56.009194648 (198h10m27.705043401s) [seg_pos={0 0}, upload=174653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:50:56.009290701 (198h10m27.705139454s) [upload=174653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:50:56.009297644 (198h10m27.705146398s) [upload=174653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:50:56.652501462 (198h10m28.348350225s) [upload=174653]: Done uploading segments... Nov 7 17:50:56.652525368 (198h10m28.348374121s) [upload=174653]: Upload finished. err: <nil> Duration: 30.002985426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.755340004s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:51:11 UTC) orbiter warning: context deadline exceeded Duration: 3.755373277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [........................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XX] duration: 30.003576537s trace: https://jaeger.dev.storj.io/trace/1fbc53a12dbf9e74 FAIL (2025-11-07 17:50:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: L6X6UB4SKUZ26ALCUIQI4SBPFVUTLTUOVVQ7AJNB62J65KL2CRBA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: Q7JFCH6WA4S76ILKCDTCURAOSIORGYQGSKW77LOMDN34ZXLIAUUA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: MXU22V4TIRABW6PTH4BZ24LWD3PAKD35GEPKXNAHKBCP7SD3LRDQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: PU34Z6OI72QR5WMD2SUQ4MGIWVVEOO27LO4NFFUJKZNTWQE74QPA) context deadline exceeded ======LOGS====== Nov 7 17:50:19.792258126 (198h9m51.488106869s) [upload=174647]: Starting upload Nov 7 17:50:19.792340613 (198h9m51.488189366s) [upload=174647]: Done starting upload Nov 7 17:50:19.792647237 (198h9m51.488496s) [upload=174647]: Uploading segments... Nov 7 17:50:19.792767827 (198h9m51.48861658s) [upload=174647]: Got next segment. Inline: false Nov 7 17:50:19.792815938 (198h9m51.488664691s) [upload=174647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:50:19.792823442 (198h9m51.488672195s) [upload=174647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:50:20.094858802 (198h9m51.790707565s) [seg_pos={0 0}, upload=174647]: Begin upload segment... Nov 7 17:50:20.095009098 (198h9m51.790857851s) [seg_pos={0 0}, upload=174647]: Done begin upload segment. Nov 7 17:50:20.095052021 (198h9m51.790900784s) [noise=true , address=45.91.116.8:4045 , piece_id=M5RBXT66J3AGX7VDL4OW5MM2U6AIJFKTJTHJXT2NGEV6VMHIQ6GA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.097104449 (198h9m51.792953202s) [noise=true , address=45.91.116.13:4153 , piece_id=K5BPFTGFZ7NWN4KLB6MVGGSUOUMGCYXIQNKV24A6SMFOOBTSWBGQ, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.098141103 (198h9m51.793989856s) [noise=true , address=77.90.31.21:4879 , piece_id=IEJLIQCGRDSEKLET7AFTCHQAI2ZCQRNXCETWU5XQGLG4O4LDPKUQ, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.098650472 (198h9m51.794499225s) [noise=true , address=45.91.116.14:4189 , piece_id=MXU22V4TIRABW6PTH4BZ24LWD3PAKD35GEPKXNAHKBCP7SD3LRDQ, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.099160855 (198h9m51.795009608s) [noise=true , address=45.91.116.5:4298 , piece_id=YOALQZLL7EPNXBJ6VQDHVQIGIAMQNFMUHOUYZVY4Y42LNLO7MQ6A, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.099633415 (198h9m51.795482168s) [noise=true , address=145.79.227.4:4186 , piece_id=6GG6EU3WGREGE2HY3QXE7BW7AY2T5ABZHOBLPKWAUFYP2GFLJ7LA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.100358885 (198h9m51.796207638s) [noise=true , address=77.90.31.10:4518 , piece_id=PU34Z6OI72QR5WMD2SUQ4MGIWVVEOO27LO4NFFUJKZNTWQE74QPA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.100842326 (198h9m51.796691089s) [noise=true , address=145.79.227.10:4403 , piece_id=NRK4YQ53TG5FC4Z2GLHH43NGXJ6I4C5ONBZLW25LKGBMDEGGQXPA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.101308454 (198h9m51.797157207s) [noise=true , address=145.79.227.11:4730 , piece_id=BLLPVUKMIACT3CXHYND2FMJP26R6YJEC5W5CH3A7HWWUPUMP4XLQ, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.101850085 (198h9m51.797698838s) [noise=true , address=45.91.116.11:4119 , piece_id=L6X6UB4SKUZ26ALCUIQI4SBPFVUTLTUOVVQ7AJNB62J65KL2CRBA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.102309860 (198h9m51.798158623s) [noise=true , address=45.91.116.4:4938 , piece_id=IFUSQMWASY7ZTAJZUA66ID76U2OVXEUDMGAEM4L56KNKHJRT4F5A, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.102780367 (198h9m51.79862912s) [noise=true , address=45.91.116.6:4973 , piece_id=CXIZS2SAERPOXA4PLPNUA4AYYTUFKEDETFSV6EAI5VXVF6E5C2DQ, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.103280589 (198h9m51.799129342s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7JFCH6WA4S76ILKCDTCURAOSIORGYQGSKW77LOMDN34ZXLIAUUA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.103789639 (198h9m51.799638402s) [noise=true , address=77.90.31.17:4770 , piece_id=5APYSIYXCJ5IBZX6O55CAMECRKFXUWMWPDSJWQQYREWR2RWKXY3Q, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.104281475 (198h9m51.800130228s) [noise=true , address=45.91.116.10:4081 , piece_id=N5QDAJMYSPFKJUFLQISSKCECEDPT4HAU723S6G4F2US2FINYSJRA, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:20.104759626 (198h9m51.800608379s) [noise=true , address=77.90.31.14:4661 , piece_id=7U5ZTIXT2PLA5RMIJIU5EEZ3GO74CJG766IV6XDBLZDBP6FFEOSQ, seg_pos={0 0}, upload=174647]: Uploading piece... Nov 7 17:50:21.938066956 (198h9m53.633915709s) [upload=174647]: Next returned nil segment Nov 7 17:50:21.938086364 (198h9m53.633935117s) [upload=174647]: Waiting for error group managing segments... Nov 7 17:50:22.234088996 (198h9m53.929937749s) [noise=true , address=145.79.227.4:4186 , piece_id=6GG6EU3WGREGE2HY3QXE7BW7AY2T5ABZHOBLPKWAUFYP2GFLJ7LA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.334401202 (198h9m54.030249955s) [noise=true , address=45.91.116.6:4973 , piece_id=CXIZS2SAERPOXA4PLPNUA4AYYTUFKEDETFSV6EAI5VXVF6E5C2DQ, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.344185603 (198h9m54.040034356s) [noise=true , address=77.90.31.17:4770 , piece_id=5APYSIYXCJ5IBZX6O55CAMECRKFXUWMWPDSJWQQYREWR2RWKXY3Q, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.350317710 (198h9m54.046166463s) [noise=true , address=45.91.116.14:4189 , piece_id=MXU22V4TIRABW6PTH4BZ24LWD3PAKD35GEPKXNAHKBCP7SD3LRDQ, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.353212232 (198h9m54.049060985s) [noise=true , address=145.79.227.10:4403 , piece_id=NRK4YQ53TG5FC4Z2GLHH43NGXJ6I4C5ONBZLW25LKGBMDEGGQXPA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.356740189 (198h9m54.052588942s) [noise=true , address=77.90.31.14:4661 , piece_id=7U5ZTIXT2PLA5RMIJIU5EEZ3GO74CJG766IV6XDBLZDBP6FFEOSQ, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.358343372 (198h9m54.054192125s) [noise=true , address=45.91.116.8:4045 , piece_id=M5RBXT66J3AGX7VDL4OW5MM2U6AIJFKTJTHJXT2NGEV6VMHIQ6GA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.380257602 (198h9m54.076106365s) [noise=true , address=45.91.116.10:4081 , piece_id=N5QDAJMYSPFKJUFLQISSKCECEDPT4HAU723S6G4F2US2FINYSJRA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.397900487 (198h9m54.09374923s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7JFCH6WA4S76ILKCDTCURAOSIORGYQGSKW77LOMDN34ZXLIAUUA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.400251084 (198h9m54.096099837s) [noise=true , address=77.90.31.21:4879 , piece_id=IEJLIQCGRDSEKLET7AFTCHQAI2ZCQRNXCETWU5XQGLG4O4LDPKUQ, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.419250583 (198h9m54.115099336s) [noise=true , address=145.79.227.11:4730 , piece_id=BLLPVUKMIACT3CXHYND2FMJP26R6YJEC5W5CH3A7HWWUPUMP4XLQ, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:22.574318265 (198h9m54.270167018s) [noise=true , address=77.90.31.10:4518 , piece_id=PU34Z6OI72QR5WMD2SUQ4MGIWVVEOO27LO4NFFUJKZNTWQE74QPA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:23.058800943 (198h9m54.754649696s) [noise=true , address=45.91.116.11:4119 , piece_id=L6X6UB4SKUZ26ALCUIQI4SBPFVUTLTUOVVQ7AJNB62J65KL2CRBA, seg_pos={0 0}, upload=174647]: Done uploading piece. err: <nil> Nov 7 17:50:23.058829949 (198h9m54.754678702s) [seg_pos={0 0}, upload=174647]: Segment reached optimal threshold of 13 pieces. Nov 7 17:50:23.059534099 (198h9m54.755382872s) [noise=true , address=45.91.116.4:4938 , piece_id=IFUSQMWASY7ZTAJZUA66ID76U2OVXEUDMGAEM4L56KNKHJRT4F5A, seg_pos={0 0}, upload=174647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:23.059547846 (198h9m54.755396609s) [noise=true , address=45.91.116.4:4938 , piece_id=IFUSQMWASY7ZTAJZUA66ID76U2OVXEUDMGAEM4L56KNKHJRT4F5A, seg_pos={0 0}, upload=174647]: Long tail cancellation due to optimal threshold Nov 7 17:50:23.059945483 (198h9m54.755794236s) [noise=true , address=45.91.116.5:4298 , piece_id=YOALQZLL7EPNXBJ6VQDHVQIGIAMQNFMUHOUYZVY4Y42LNLO7MQ6A, seg_pos={0 0}, upload=174647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:23.059954309 (198h9m54.755803052s) [noise=true , address=45.91.116.5:4298 , piece_id=YOALQZLL7EPNXBJ6VQDHVQIGIAMQNFMUHOUYZVY4Y42LNLO7MQ6A, seg_pos={0 0}, upload=174647]: Long tail cancellation due to optimal threshold Nov 7 17:50:23.060156104 (198h9m54.756004857s) [noise=true , address=45.91.116.13:4153 , piece_id=K5BPFTGFZ7NWN4KLB6MVGGSUOUMGCYXIQNKV24A6SMFOOBTSWBGQ, seg_pos={0 0}, upload=174647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 17:50:23.060162916 (198h9m54.756011669s) [noise=true , address=45.91.116.13:4153 , piece_id=K5BPFTGFZ7NWN4KLB6MVGGSUOUMGCYXIQNKV24A6SMFOOBTSWBGQ, seg_pos={0 0}, upload=174647]: Long tail cancellation due to optimal threshold Nov 7 17:50:23.060180931 (198h9m54.756029684s) [seg_pos={0 0}, upload=174647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:50:23.060282755 (198h9m54.756131508s) [upload=174647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:50:23.060290209 (198h9m54.756138962s) [upload=174647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:50:23.385612122 (198h9m55.081460885s) [upload=174647]: Done uploading segments... Nov 7 17:50:23.385639785 (198h9m55.081488538s) [upload=174647]: Upload finished. err: <nil> Duration: 30.003591003s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.74621269s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:50:43 UTC) orbiter warning: context deadline exceeded Duration: 3.746241003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.........................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..X] duration: 30.003768676s trace: https://jaeger.dev.storj.io/trace/17aa20e73719b260 FAIL (2025-11-07 17:50:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 7 17:49:49.563019888 (198h9m21.258868641s) [upload=174638]: Starting upload Nov 7 17:49:49.563127793 (198h9m21.258976546s) [upload=174638]: Done starting upload Nov 7 17:49:49.563382357 (198h9m21.25923111s) [upload=174638]: Uploading segments... Nov 7 17:49:49.563492367 (198h9m21.25934112s) [upload=174638]: Got next segment. Inline: false Nov 7 17:49:49.564354829 (198h9m21.260203582s) [upload=174638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:49:49.564361973 (198h9m21.260210716s) [upload=174638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:49:50.386016640 (198h9m22.081865393s) [seg_pos={0 0}, upload=174638]: Begin upload segment... Nov 7 17:49:50.386083868 (198h9m22.081932622s) [seg_pos={0 0}, upload=174638]: Done begin upload segment. Nov 7 17:49:50.386121340 (198h9m22.081970093s) [noise=true , address=45.91.116.11:4119 , piece_id=3SAJ2LUU46LTJCHBTRW5XSMXS2JK33DRUJUBN6WPO3PG5DOKDAMQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.386148390 (198h9m22.081997144s) [noise=true , address=45.91.116.5:4297 , piece_id=DB7KSNKDWMOOWAARCSDITTNUL3MNXYKJDGQ44FHT5MCALDZ4Y5NA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.386792908 (198h9m22.082641671s) [noise=true , address=145.79.227.18:4005 , piece_id=6DYM5RQL6UWRCE5GUD3FEGI33MZSFQRKK2HBGIBFZIOX2L3LQGSQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.387211165 (198h9m22.083059938s) [noise=true , address=77.90.31.12:4590 , piece_id=T2IRBS4HBUO6JJZGRPBRBFPCYJ5Z4O4GQZY25VUVIGVC2DH73PNQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.387833500 (198h9m22.083682253s) [noise=true , address=145.79.227.27:4690 , piece_id=CDDBLCV3TNWYBWG5WQZMP6XAZNDISSRGVULQKQ6UJC3I3F4JUL7A, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.388674902 (198h9m22.084523655s) [noise=true , address=145.79.227.22:4509 , piece_id=VXKARAO2TVGQPGYTE26DHQPXU4JRIZKCRCZ2EXVQMP4IZ4LKYAAA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.389123917 (198h9m22.08497268s) [noise=true , address=145.79.227.30:4763 , piece_id=3KZQHJWOEJ5JJD6LWZ3A6YRHOCRKERZPVUVPCFPPQTQFWUUONOAA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.389582179 (198h9m22.085430932s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXYEPZAUAN636RTDBSPG7QISGELSNGIHWSMKB2GF6DXH6KLE2SA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.389620492 (198h9m22.085469245s) [noise=true , address=145.79.227.13:4803 , piece_id=3X3DJ7HYVK7DUKU2I7WF2GLW5NIWBMGNOPMUBXV47Y6ALPZTL5YA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.390106287 (198h9m22.08595504s) [noise=true , address=145.79.227.8:4330 , piece_id=NLCDRIGYR6QDMP4UMPMQZZURIQVV3R2R777NTVK7Z7GML7DF3IXQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.390562275 (198h9m22.086411039s) [noise=true , address=45.91.116.14:4191 , piece_id=VSVEK4MNOX4UWZZBSI4LAO2MTNWGCT5IWRDK27IG2WYHZEADG6CQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.390994770 (198h9m22.086843513s) [noise=true , address=145.79.227.25:4618 , piece_id=6MGWFHEP7TRD3QTI6RAYKDNPA2VVF3GRYM5UTQQQFAYAZF2XNILA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.391090060 (198h9m22.086938813s) [noise=true , address=145.79.227.4:4186 , piece_id=NTXMCTD4KXOU2X34RZBAOCBCJG7NQ4H6HI323KWYV2O6JI6VFBYQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.391668803 (198h9m22.087517556s) [noise=true , address=77.90.31.16:4734 , piece_id=RAIZPQZKPCQNZ4BF5RK66N3O7QQHCPOQ64ELBVYRFV2LWI6XCQ2A, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.391966079 (198h9m22.087814832s) [noise=true , address=145.79.227.10:4402 , piece_id=SJ6ZTLOPIG5GREJ4JUKLOI5JVLPDNYUTTWVYFDFENB46SQY2VMTQ, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:50.392147264 (198h9m22.087996017s) [noise=true , address=145.79.227.17:4972 , piece_id=6W7BDDKDQPG47BGN7OTFZENGLNB23JFUIBJTJMY7HAUOH7IBZ4CA, seg_pos={0 0}, upload=174638]: Uploading piece... Nov 7 17:49:52.085372951 (198h9m23.781221704s) [upload=174638]: Next returned nil segment Nov 7 17:49:52.085393941 (198h9m23.781242724s) [upload=174638]: Waiting for error group managing segments... Nov 7 17:49:52.486681287 (198h9m24.18253004s) [noise=true , address=145.79.227.8:4330 , piece_id=NLCDRIGYR6QDMP4UMPMQZZURIQVV3R2R777NTVK7Z7GML7DF3IXQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.486773142 (198h9m24.182621905s) [noise=true , address=145.79.227.27:4690 , piece_id=CDDBLCV3TNWYBWG5WQZMP6XAZNDISSRGVULQKQ6UJC3I3F4JUL7A, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.494312178 (198h9m24.190160931s) [noise=true , address=145.79.227.22:4509 , piece_id=VXKARAO2TVGQPGYTE26DHQPXU4JRIZKCRCZ2EXVQMP4IZ4LKYAAA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.555198709 (198h9m24.251047462s) [noise=true , address=145.79.227.17:4972 , piece_id=6W7BDDKDQPG47BGN7OTFZENGLNB23JFUIBJTJMY7HAUOH7IBZ4CA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.563652808 (198h9m24.259501561s) [noise=true , address=145.79.227.4:4186 , piece_id=NTXMCTD4KXOU2X34RZBAOCBCJG7NQ4H6HI323KWYV2O6JI6VFBYQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.625723685 (198h9m24.321572438s) [noise=true , address=45.91.116.11:4119 , piece_id=3SAJ2LUU46LTJCHBTRW5XSMXS2JK33DRUJUBN6WPO3PG5DOKDAMQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.654078277 (198h9m24.34992703s) [noise=true , address=45.91.116.5:4297 , piece_id=DB7KSNKDWMOOWAARCSDITTNUL3MNXYKJDGQ44FHT5MCALDZ4Y5NA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.685697287 (198h9m24.38154604s) [noise=true , address=45.91.116.14:4191 , piece_id=VSVEK4MNOX4UWZZBSI4LAO2MTNWGCT5IWRDK27IG2WYHZEADG6CQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.692831301 (198h9m24.388680054s) [noise=true , address=145.79.227.10:4402 , piece_id=SJ6ZTLOPIG5GREJ4JUKLOI5JVLPDNYUTTWVYFDFENB46SQY2VMTQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.692989362 (198h9m24.388838105s) [noise=true , address=145.79.227.13:4803 , piece_id=3X3DJ7HYVK7DUKU2I7WF2GLW5NIWBMGNOPMUBXV47Y6ALPZTL5YA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.705026103 (198h9m24.400874856s) [noise=true , address=77.90.31.12:4590 , piece_id=T2IRBS4HBUO6JJZGRPBRBFPCYJ5Z4O4GQZY25VUVIGVC2DH73PNQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.765774952 (198h9m24.461623705s) [noise=true , address=145.79.227.18:4005 , piece_id=6DYM5RQL6UWRCE5GUD3FEGI33MZSFQRKK2HBGIBFZIOX2L3LQGSQ, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.834339794 (198h9m24.530188547s) [noise=true , address=77.90.31.16:4734 , piece_id=RAIZPQZKPCQNZ4BF5RK66N3O7QQHCPOQ64ELBVYRFV2LWI6XCQ2A, seg_pos={0 0}, upload=174638]: Done uploading piece. err: <nil> Nov 7 17:49:52.834366224 (198h9m24.530214977s) [seg_pos={0 0}, upload=174638]: Segment reached optimal threshold of 13 pieces. Nov 7 17:49:52.834605570 (198h9m24.530454333s) [noise=true , address=145.79.227.25:4618 , piece_id=6MGWFHEP7TRD3QTI6RAYKDNPA2VVF3GRYM5UTQQQFAYAZF2XNILA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Nov 7 17:49:52.834614467 (198h9m24.53046322s) [noise=true , address=145.79.227.25:4618 , piece_id=6MGWFHEP7TRD3QTI6RAYKDNPA2VVF3GRYM5UTQQQFAYAZF2XNILA, seg_pos={0 0}, upload=174638]: Long tail cancellation due to optimal threshold Nov 7 17:49:52.834743002 (198h9m24.530591745s) [noise=true , address=145.79.227.30:4763 , piece_id=3KZQHJWOEJ5JJD6LWZ3A6YRHOCRKERZPVUVPCFPPQTQFWUUONOAA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; context canceled; EOF Nov 7 17:49:52.834754734 (198h9m24.530603477s) [noise=true , address=145.79.227.30:4763 , piece_id=3KZQHJWOEJ5JJD6LWZ3A6YRHOCRKERZPVUVPCFPPQTQFWUUONOAA, seg_pos={0 0}, upload=174638]: Long tail cancellation due to optimal threshold Nov 7 17:49:52.835099721 (198h9m24.530948464s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXYEPZAUAN636RTDBSPG7QISGELSNGIHWSMKB2GF6DXH6KLE2SA, seg_pos={0 0}, upload=174638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 7 17:49:52.835107285 (198h9m24.530956038s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXYEPZAUAN636RTDBSPG7QISGELSNGIHWSMKB2GF6DXH6KLE2SA, seg_pos={0 0}, upload=174638]: Long tail cancellation due to optimal threshold Nov 7 17:49:52.835145528 (198h9m24.530994281s) [seg_pos={0 0}, upload=174638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 17:49:52.835271678 (198h9m24.531120431s) [upload=174638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:49:52.835278902 (198h9m24.531127645s) [upload=174638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:49:53.147475812 (198h9m24.843324565s) [upload=174638]: Done uploading segments... Nov 7 17:49:53.147501992 (198h9m24.843350746s) [upload=174638]: Upload finished. err: <nil> Duration: 30.003791489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.430971472s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:50:16 UTC) orbiter warning: context deadline exceeded Duration: 3.431011519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.532874676s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 17:49:12 UTC) orbiter warning: context deadline exceeded Duration: 3.532904623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..........................XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..] duration: 22.606763271s trace: https://jaeger.dev.storj.io/trace/15b69016dd8c6293 FAIL (2025-11-07 17:48:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: DJP4WGIX7QI7QZAZOXBEOKVZDRI2QJJLTLUIZSFRKWW5XYT6I57A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: CMN6THJ3XXZGGNMRPZAWGQT32QXWAQOE5GJCF4BZXK4XZVFFJZ3A) context deadline exceeded ======LOGS====== Nov 7 17:46:58.572072276 (198h6m30.267921029s) [upload=174599]: Starting upload Nov 7 17:46:58.572199438 (198h6m30.268048181s) [upload=174599]: Done starting upload Nov 7 17:46:58.572373961 (198h6m30.268222714s) [upload=174599]: Uploading segments... Nov 7 17:46:58.572747312 (198h6m30.268596065s) [upload=174599]: Got next segment. Inline: false Nov 7 17:46:58.572816934 (198h6m30.268665687s) [upload=174599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 17:46:58.572824799 (198h6m30.268673552s) [upload=174599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 17:46:58.829359726 (198h6m30.525208479s) [seg_pos={0 0}, upload=174599]: Begin upload segment... Nov 7 17:46:58.829420603 (198h6m30.525269356s) [seg_pos={0 0}, upload=174599]: Done begin upload segment. Nov 7 17:46:58.829454798 (198h6m30.525303541s) [noise=true , address=145.79.227.31:4799 , piece_id=5ESMOWEC3CKXQQ7MB6JBJUOV7KJULPNW67SCYGUSVRMMQGWQGJ3A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.829486788 (198h6m30.525335541s) [noise=true , address=145.79.227.15:4874 , piece_id=C4I25KBPFRJM5SK3KXQASTG73F27ODT4PP5NXYEOUOLFW74MMSUQ, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.830183114 (198h6m30.526031868s) [noise=true , address=145.79.227.20:4439 , piece_id=LWNYK6YAHY7Q3EYEVCF6KADRUQBUE3Q4RZP3SLSOIBMYIV55XG5A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.830783267 (198h6m30.52663201s) [noise=true , address=45.91.116.6:4974 , piece_id=XW7THJXPUYDIBAVJWAT4KTDVBFUT7DGOKTZ6OAGQMKUS6AN3PY3Q, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.832108721 (198h6m30.527957474s) [noise=true , address=77.90.31.10:4520 , piece_id=5FPTETMUWJ353CIULDDVXK3BPRI4WW3H3IKI5DHT5LDGWQXRMXHA, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.833258060 (198h6m30.529106813s) [noise=true , address=145.79.227.32:4836 , piece_id=QVFWZ6HKP5FUSBN7ZWJYEKSRDKERILAC5QEFILUQH6XUGBWZUC3A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.833288788 (198h6m30.529137541s) [noise=true , address=77.90.31.13:4628 , piece_id=G5PHD3HQE2MJ5ZT2P242NX6Z7KLYJCEXLZTAQ4H4RMV4PNV4X7MQ, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.833665926 (198h6m30.52951468s) [noise=true , address=45.91.116.12:4226 , piece_id=CMN6THJ3XXZGGNMRPZAWGQT32QXWAQOE5GJCF4BZXK4XZVFFJZ3A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.834582251 (198h6m30.530430994s) [noise=true , address=145.79.227.14:4837 , piece_id=VZ22E3X7QAOXWAIRDDIT7XROPPFSQQLIBDTFFAC4VHNEQZAR73IA, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.836307076 (198h6m30.532155829s) [noise=true , address=45.91.116.13:4154 , piece_id=EPHLLGAL7NYHH7O7FHAZR3PU2PFYKFQYM4KA4Q6WXQKAPAT4ZD4A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.837027288 (198h6m30.532876041s) [noise=true , address=145.79.227.16:4909 , piece_id=II6W7LAVNKRLE3I2B5ZV7JAUIYDO3VT2AU53GWMKI36PQ3DTEY5A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.838362339 (198h6m30.534211142s) [noise=true , address=145.79.227.10:4402 , piece_id=XBOKW2YXJJE2AXFT7U5GC7AWPVXIJ33DFYH6TFJ7XXPKX2SR23PQ, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.838874455 (198h6m30.534723208s) [noise=true , address=45.91.116.9:4263 , piece_id=DJP4WGIX7QI7QZAZOXBEOKVZDRI2QJJLTLUIZSFRKWW5XYT6I57A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.840050604 (198h6m30.535899357s) [noise=true , address=45.91.116.14:4189 , piece_id=WG56QK2C7KJJQWPQBFIXCITQLX5BTMNGPTB25A6N4LLJFQXSYJ4A, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.840925410 (198h6m30.536774163s) [noise=true , address=145.79.227.4:4186 , piece_id=UQBXGZAM7AU56XMUNN2SB6GOUMBOR73TPBTYSDJRF2BJR2HNC56Q, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:46:58.842933303 (198h6m30.538782056s) [noise=true , address=145.79.227.25:4618 , piece_id=VJ3RLPFQBF7RNSSWXFGMQJOGZSLW6X6CQ265ESBHLGL4KPZC64ZA, seg_pos={0 0}, upload=174599]: Uploading piece... Nov 7 17:47:00.689155355 (198h6m32.385004118s) [upload=174599]: Next returned nil segment Nov 7 17:47:00.689201864 (198h6m32.385050617s) [upload=174599]: Waiting for error group managing segments... Nov 7 17:47:01.067551984 (198h6m32.763400737s) [noise=true , address=45.91.116.14:4189 , piece_id=WG56QK2C7KJJQWPQBFIXCITQLX5BTMNGPTB25A6N4LLJFQXSYJ4A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.081733258 (198h6m32.777582011s) [noise=true , address=145.79.227.4:4186 , piece_id=UQBXGZAM7AU56XMUNN2SB6GOUMBOR73TPBTYSDJRF2BJR2HNC56Q, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.089683718 (198h6m32.785532471s) [noise=true , address=145.79.227.20:4439 , piece_id=LWNYK6YAHY7Q3EYEVCF6KADRUQBUE3Q4RZP3SLSOIBMYIV55XG5A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.094645556 (198h6m32.790494309s) [noise=true , address=145.79.227.31:4799 , piece_id=5ESMOWEC3CKXQQ7MB6JBJUOV7KJULPNW67SCYGUSVRMMQGWQGJ3A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.103460060 (198h6m32.799308813s) [noise=true , address=145.79.227.15:4874 , piece_id=C4I25KBPFRJM5SK3KXQASTG73F27ODT4PP5NXYEOUOLFW74MMSUQ, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.109099339 (198h6m32.804948083s) [noise=true , address=45.91.116.12:4226 , piece_id=CMN6THJ3XXZGGNMRPZAWGQT32QXWAQOE5GJCF4BZXK4XZVFFJZ3A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.127364421 (198h6m32.823213194s) [noise=true , address=77.90.31.13:4628 , piece_id=G5PHD3HQE2MJ5ZT2P242NX6Z7KLYJCEXLZTAQ4H4RMV4PNV4X7MQ, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.127940707 (198h6m32.823789471s) [noise=true , address=145.79.227.16:4909 , piece_id=II6W7LAVNKRLE3I2B5ZV7JAUIYDO3VT2AU53GWMKI36PQ3DTEY5A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.145842997 (198h6m32.84169175s) [noise=true , address=45.91.116.9:4263 , piece_id=DJP4WGIX7QI7QZAZOXBEOKVZDRI2QJJLTLUIZSFRKWW5XYT6I57A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.148585939 (198h6m32.844434703s) [noise=true , address=145.79.227.32:4836 , piece_id=QVFWZ6HKP5FUSBN7ZWJYEKSRDKERILAC5QEFILUQH6XUGBWZUC3A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.149069100 (198h6m32.844917853s) [noise=true , address=145.79.227.25:4618 , piece_id=VJ3RLPFQBF7RNSSWXFGMQJOGZSLW6X6CQ265ESBHLGL4KPZC64ZA, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.159015721 (198h6m32.854864484s) [noise=true , address=77.90.31.10:4520 , piece_id=5FPTETMUWJ353CIULDDVXK3BPRI4WW3H3IKI5DHT5LDGWQXRMXHA, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.175592636 (198h6m32.871441389s) [noise=true , address=145.79.227.14:4837 , piece_id=VZ22E3X7QAOXWAIRDDIT7XROPPFSQQLIBDTFFAC4VHNEQZAR73IA, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.175623685 (198h6m32.871472438s) [seg_pos={0 0}, upload=174599]: Segment reached optimal threshold of 13 pieces. Nov 7 17:47:01.175804239 (198h6m32.871652992s) [noise=true , address=145.79.227.10:4402 , piece_id=XBOKW2YXJJE2AXFT7U5GC7AWPVXIJ33DFYH6TFJ7XXPKX2SR23PQ, seg_pos={0 0}, upload=174599]: Done uploading piece. err: <nil> Nov 7 17:47:01.176074484 (198h6m32.871923257s) [noise=true , address=45.91.116.6:4974 , piece_id=XW7THJXPUYDIBAVJWAT4KTDVBFUT7DGOKTZ6OAGQMKUS6AN3PY3Q, seg_pos={0 0}, upload=174599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:53924->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:53924->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53924->45.91.116.6:4974: use of closed network connection Nov 7 17:47:01.176084834 (198h6m32.871933587s) [noise=true , address=45.91.116.6:4974 , piece_id=XW7THJXPUYDIBAVJWAT4KTDVBFUT7DGOKTZ6OAGQMKUS6AN3PY3Q, seg_pos={0 0}, upload=174599]: Long tail cancellation due to optimal threshold Nov 7 17:47:01.176138315 (198h6m32.871987058s) [noise=true , address=45.91.116.13:4154 , piece_id=EPHLLGAL7NYHH7O7FHAZR3PU2PFYKFQYM4KA4Q6WXQKAPAT4ZD4A, seg_pos={0 0}, upload=174599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Nov 7 17:47:01.176149286 (198h6m32.871998029s) [noise=true , address=45.91.116.13:4154 , piece_id=EPHLLGAL7NYHH7O7FHAZR3PU2PFYKFQYM4KA4Q6WXQKAPAT4ZD4A, seg_pos={0 0}, upload=174599]: Long tail cancellation due to optimal threshold Nov 7 17:47:01.176263755 (198h6m32.872112508s) [seg_pos={0 0}, upload=174599]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 7 17:47:01.176390065 (198h6m32.872238818s) [upload=174599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 17:47:01.176397650 (198h6m32.872246403s) [upload=174599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 17:47:01.472175062 (198h6m33.168023835s) [upload=174599]: Done uploading segments... Nov 7 17:47:01.472201783 (198h6m33.168050536s) [upload=174599]: Upload finished. err: <nil> Duration: 22.606792467s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)