Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.X.XX.X] duration: 5.001750713s trace: https://jaeger.dev.storj.io/trace/5daef18b73950654 FAIL (2025-12-16 15:57:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001782443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.............................................X..X..........] duration: 5.763132152s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:57:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.763158771s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.X.XX.] duration: 4.798309685s trace: https://jaeger.dev.storj.io/trace/eb5a66511d8c1b1 FAIL (2025-12-16 15:56:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.798337799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.............................................X..X.........] duration: 5.761938959s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:56:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.761958006s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.X.XX] duration: 5.000781958s trace: https://jaeger.dev.storj.io/trace/eb5a66511d8c1b1 FAIL (2025-12-16 15:56:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000804129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.............................................X..X........] duration: 5.751397666s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:56:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.751427413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.X.X] duration: 5.005382458s trace: https://jaeger.dev.storj.io/trace/2c0207727be45449 FAIL (2025-12-16 15:56:15 UTC) orbiter warning: context deadline exceeded Duration: 5.005412775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.............................................X..X.......] duration: 5.743171803s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:56:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.743201479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............................................X..X......] duration: 5.964038861s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:55:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.964065111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.X.] duration: 3.656786683s trace: https://jaeger.dev.storj.io/trace/19bb1b8551be5593 FAIL (2025-12-16 15:55:18 UTC) orbiter warning: context deadline exceeded Duration: 3.656819344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............................................X..X.....] duration: 5.922252944s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:55:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.922270138s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.X] duration: 5.001086406s trace: https://jaeger.dev.storj.io/trace/19bb1b8551be5593 FAIL (2025-12-16 15:54:24 UTC) orbiter warning: context deadline exceeded Duration: 5.001114822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............................................X..X....] duration: 6.17550457s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:54:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 6.175530639s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X.] duration: 5.002385631s trace: https://jaeger.dev.storj.io/trace/27282b1066d33a6f FAIL (2025-12-16 15:53:49 UTC) orbiter warning: context deadline exceeded Duration: 5.002411209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............................................X..X...] duration: 7.272353224s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:53:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 7.272377119s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..X] duration: 5.001339479s trace: https://jaeger.dev.storj.io/trace/27282b1066d33a6f FAIL (2025-12-16 15:53:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001363755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.............................................X..X..] duration: 5.702833602s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.70293202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.............................................X..X.] duration: 5.803247087s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 5.803275621s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX..] duration: 3.968238659s trace: https://jaeger.dev.storj.io/trace/917bfe20c208d53 FAIL (2025-12-16 15:52:40 UTC) orbiter warning: context deadline exceeded Duration: 3.968251724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.............................................X..X] duration: 30.00267389s trace: https://jaeger.dev.storj.io/trace/5449a2f9afc8e48d FAIL (2025-12-16 15:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:43.513611062 (264h22m43.384694108s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:51:43.759386599 (264h22m43.630469634s) [seg_pos={0 0}, upload=231766]: Begin upload segment... Dec 16 15:51:43.759443748 (264h22m43.630526783s) [seg_pos={0 0}, upload=231766]: Done begin upload segment. Dec 16 15:51:43.759477523 (264h22m43.630560558s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.759578916 (264h22m43.630661951s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760084780 (264h22m43.631167815s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760578259 (264h22m43.631661294s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.760968552 (264h22m43.632051587s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761336642 (264h22m43.632419687s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.761720965 (264h22m43.632804s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762229713 (264h22m43.633312758s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.762684459 (264h22m43.633767504s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.763623778 (264h22m43.634706813s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.764037746 (264h22m43.635120781s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.765151377 (264h22m43.636234422s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.766336012 (264h22m43.637419058s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.768093680 (264h22m43.639176715s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.769028108 (264h22m43.640111173s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:43.770327173 (264h22m43.641410208s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Uploading piece... Dec 16 15:51:45.368827227 (264h22m45.239910262s) [upload=231766]: Next returned nil segment Dec 16 15:51:45.368846855 (264h22m45.2399299s) [upload=231766]: Waiting for error group managing segments... Dec 16 15:51:45.750686672 (264h22m45.621769707s) [noise=true , address=145.79.227.45:4231 , piece_id=D64GNDCAODZLT2Q4DEEIBOOGV257NREPTLHYMBKYAUOH2KPC2OEA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.752190084 (264h22m45.623273119s) [noise=true , address=145.79.227.38:4979 , piece_id=CT2GTGOI32W53KX2MAC2ADDWSZKWPRA3JOVXULATUW22GVWRY6JA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.754484634 (264h22m45.625567679s) [noise=true , address=145.79.227.47:4267 , piece_id=NPXBNOK4GQAO6VSZKXS3W7WCEYR62AXAZYMFFO26XG6NX7Q3FLZA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.758008994 (264h22m45.629092029s) [noise=true , address=145.79.227.52:4446 , piece_id=6NB2EWHVDOJ7BSXXCPEDHSCPZKTNHTSFTEN5US2AHBT7OWIYEEBQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.777717934 (264h22m45.648800969s) [noise=true , address=145.79.227.49:4339 , piece_id=54YCFROGSQ4A7ZP3XDMGPEM3QJX3AII2MKAJTQZKP73WZUFEOQOQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.784827190 (264h22m45.655910225s) [noise=true , address=145.79.227.42:4122 , piece_id=44KGPSRTCME2YQJLGQ3NOQEWKVTTXNIDKM7XDHFUNI3UPZUNKX4Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.788774938 (264h22m45.659857973s) [noise=true , address=145.79.227.9:4365 , piece_id=OENIEIE2XBIOHMV5CJYENDLEMI7RA7CKNWC5TPXOA7OV6DHZI3HQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.824436263 (264h22m45.695519298s) [noise=true , address=145.79.227.3:4150 , piece_id=5DK57TNPJU2VTSZMKK62W7NYICXB5A5TDYUTMW2FOPRFOSXU6MJA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.835233453 (264h22m45.706316488s) [noise=true , address=145.79.227.18:4007 , piece_id=JIE42T2OBMFJ23NL5E77JTKP2DSLDJQUG52JFXOYWRUEYX6TOLXA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.848457524 (264h22m45.719540559s) [noise=true , address=145.79.227.39:4014 , piece_id=3NMNVKKPSRNFZG45G3E5T2WTFJIMGU6FGH6NEPBZHSANEBHXE22Q, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:45.999904073 (264h22m45.870987129s) [noise=true , address=45.91.116.14:4192 , piece_id=RK7CUTUCIBMMG53JGTFBPNPTNFTQVD5Q5JTOAYWECP7HFCGHO5MQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.014701300 (264h22m45.885784345s) [noise=true , address=77.90.31.18:4807 , piece_id=OO3SSFKUWBILUK5K3ATKT2KEVGD4F2VRKATSMALTWCEMTKKMP5IQ, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102364225 (264h22m45.97344726s) [noise=true , address=45.91.116.11:4120 , piece_id=RCSFRLO3JNECDCXHVOVWA43L4TLKTZIJK6NZST2B5A475LEGGBBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: <nil> Dec 16 15:51:46.102388520 (264h22m45.973471556s) [seg_pos={0 0}, upload=231766]: Segment reached optimal threshold of 13 pieces. Dec 16 15:51:46.102575497 (264h22m45.973658532s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43364->77.90.31.17:4770: use of closed network connection Dec 16 15:51:46.102581548 (264h22m45.973664594s) [noise=true , address=77.90.31.17:4770 , piece_id=DPKLQEGVJ54FKOM527SCSDOZGXYC2UGLZID6VMVYCEJMO276EKTA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102721946 (264h22m45.973804981s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->45.91.116.4:4937: use of closed network connection Dec 16 15:51:46.102750070 (264h22m45.973833105s) [noise=true , address=45.91.116.4:4937 , piece_id=S36DMH6G3HM43N6MSUF6MLO6KEHKQHJ77P6SWEESL2XLHPONLHBA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102915996 (264h22m45.973999041s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40748->45.91.116.9:4264: use of closed network connection Dec 16 15:51:46.102926505 (264h22m45.974009561s) [noise=true , address=45.91.116.9:4264 , piece_id=H34SCEPCK4NGIZN4RK5CPOQZHJNGOGC7RZZLTBVMQHYREPRVXKQA, seg_pos={0 0}, upload=231766]: Long tail cancellation due to optimal threshold Dec 16 15:51:46.102948397 (264h22m45.974031433s) [seg_pos={0 0}, upload=231766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:51:46.103052195 (264h22m45.97413523s) [upload=231766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:51:46.103057525 (264h22m45.97414056s) [upload=231766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:51:46.412045651 (264h22m46.283128696s) [upload=231766]: Done uploading segments... Dec 16 15:51:46.412064716 (264h22m46.283147751s) [upload=231766]: Upload finished. err: <nil> Duration:* 30.002696033s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX.] duration: 4.090536977s trace: https://jaeger.dev.storj.io/trace/917bfe20c208d53 FAIL (2025-12-16 15:52:08 UTC) orbiter warning: context deadline exceeded Duration: 4.090553828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XXX] duration: 5.001288235s trace: https://jaeger.dev.storj.io/trace/917bfe20c208d53 FAIL (2025-12-16 15:51:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001311329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............................................X..] duration: 5.784471699s trace: https://jaeger.dev.storj.io/trace/2c8955a88808a9c7 FAIL (2025-12-16 15:51:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ) context deadline exceeded ======LOGS====== 1735]: Uploading segments... Dec 16 15:49:57.849123937 (264h20m57.720206992s) [upload=231735]: Got next segment. Inline: false Dec 16 15:49:57.849153704 (264h20m57.720236749s) [upload=231735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:49:57.849159384 (264h20m57.720242429s) [upload=231735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:49:58.100982669 (264h20m57.972065704s) [seg_pos={0 0}, upload=231735]: Begin upload segment... Dec 16 15:49:58.101029037 (264h20m57.972112072s) [seg_pos={0 0}, upload=231735]: Done begin upload segment. Dec 16 15:49:58.101096566 (264h20m57.972179601s) [noise=true , address=145.79.227.41:4085 , piece_id=4TQXRTBJUO725YUYWHF55IXTPMJZV4BNZN6AYN5QZRG3J7IR66OA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101118829 (264h20m57.972201864s) [noise=true , address=145.79.227.33:4869 , piece_id=3BO5NKJTJHTIAUM3TIVKJPUIPKQ34ICZ2MMO4FS6TB2FTJENVLOA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101574506 (264h20m57.972657541s) [noise=true , address=145.79.227.52:4446 , piece_id=IF6ASWX3KXUJBBTO66Q6VWWDM57UJETZTKBCGUW6QZW2JR7FLKTA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101890688 (264h20m57.972973733s) [noise=true , address=145.79.227.39:4014 , piece_id=6R2CLDIMLKPUTO5VLRJQLKHA6COBYH6EEQCEN6JDSIZOIWXHZTTA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102264310 (264h20m57.973347355s) [noise=true , address=145.79.227.3:4150 , piece_id=7GPTKGQYPELZMSLSFIZEEN7TNORVIU26Q27JBLF553YCYWYTMZYQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102564922 (264h20m57.973647967s) [noise=true , address=145.79.227.20:4437 , piece_id=4XGSBK3RWNOCSYOHUN4SRQ3JZ5LHC53RDH5MMF2RKKEF6Q5PBSUA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102856147 (264h20m57.973939192s) [noise=true , address=145.79.227.17:4970 , piece_id=LD3KKLNQEQKPQN36KNJA24JU3VFFMP6TJSXD2OY62KBSE3P4ZOKA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.103234788 (264h20m57.974317833s) [noise=true , address=145.79.227.46:4603 , piece_id=KNHAOIYBT4Y4AI7ZX37EWWO2PPWI7JDJUPXYFDCL3BPLNESCIDQQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.103630621 (264h20m57.974713656s) [noise=true , address=145.79.227.40:4050 , piece_id=KCTVDTVNAVOHKTPYJZVWMEVTR2A32FFUEGIBJTA77ZKDCIKJIO5A, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.105265925 (264h20m57.97634897s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.107092713 (264h20m57.978175748s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.108896789 (264h20m57.979979834s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.110314729 (264h20m57.981397774s) [noise=true , address=45.91.116.13:4156 , piece_id=RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.111559820 (264h20m57.982642855s) [noise=true , address=145.79.227.27:4690 , piece_id=5HUGLIYU3WRWBTBZAMHK3DCOJENPCDW3X63R7F47LLJMB6FB4RUA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.112108515 (264h20m57.98319155s) [noise=true , address=145.79.227.7:4293 , piece_id=UTII7KLLYGVDKKI6C2JFP5QL45KY2YYHK5ZAKWXLRE3YOTWQ32AA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.113391828 (264h20m57.984474863s) [noise=true , address=77.90.31.17:4773 , piece_id=FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:59.783147534 (264h20m59.654230569s) [upload=231735]: Next returned nil segment Dec 16 15:49:59.783160720 (264h20m59.654243765s) [upload=231735]: Waiting for error group managing segments... Dec 16 15:50:00.093340434 (264h20m59.96442348s) [noise=true , address=145.79.227.41:4085 , piece_id=4TQXRTBJUO725YUYWHF55IXTPMJZV4BNZN6AYN5QZRG3J7IR66OA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.099886569 (264h20m59.970969604s) [noise=true , address=145.79.227.52:4446 , piece_id=IF6ASWX3KXUJBBTO66Q6VWWDM57UJETZTKBCGUW6QZW2JR7FLKTA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.100217729 (264h20m59.971300775s) [noise=true , address=145.79.227.17:4970 , piece_id=LD3KKLNQEQKPQN36KNJA24JU3VFFMP6TJSXD2OY62KBSE3P4ZOKA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.102396048 (264h20m59.973479123s) [noise=true , address=145.79.227.27:4690 , piece_id=5HUGLIYU3WRWBTBZAMHK3DCOJENPCDW3X63R7F47LLJMB6FB4RUA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.107923433 (264h20m59.979006478s) [noise=true , address=145.79.227.39:4014 , piece_id=6R2CLDIMLKPUTO5VLRJQLKHA6COBYH6EEQCEN6JDSIZOIWXHZTTA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.113831112 (264h20m59.984914157s) [noise=true , address=145.79.227.20:4437 , piece_id=4XGSBK3RWNOCSYOHUN4SRQ3JZ5LHC53RDH5MMF2RKKEF6Q5PBSUA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.119556885 (264h20m59.99063993s) [noise=true , address=145.79.227.3:4150 , piece_id=7GPTKGQYPELZMSLSFIZEEN7TNORVIU26Q27JBLF553YCYWYTMZYQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.123421494 (264h20m59.994504529s) [noise=true , address=145.79.227.7:4293 , piece_id=UTII7KLLYGVDKKI6C2JFP5QL45KY2YYHK5ZAKWXLRE3YOTWQ32AA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.125340808 (264h20m59.996423853s) [noise=true , address=145.79.227.40:4050 , piece_id=KCTVDTVNAVOHKTPYJZVWMEVTR2A32FFUEGIBJTA77ZKDCIKJIO5A, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.162453147 (264h21m0.033536192s) [noise=true , address=145.79.227.46:4603 , piece_id=KNHAOIYBT4Y4AI7ZX37EWWO2PPWI7JDJUPXYFDCL3BPLNESCIDQQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.415415770 (264h21m0.286498825s) [noise=true , address=77.90.31.17:4773 , piece_id=FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.460990852 (264h21m0.332073887s) [noise=true , address=145.79.227.33:4869 , piece_id=3BO5NKJTJHTIAUM3TIVKJPUIPKQ34ICZ2MMO4FS6TB2FTJENVLOA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.695944363 (264h21m0.567027399s) [noise=true , address=45.91.116.13:4156 , piece_id=RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.695971876 (264h21m0.567054911s) [seg_pos={0 0}, upload=231735]: Segment reached optimal threshold of 13 pieces. Dec 16 15:50:00.696268691 (264h21m0.567351736s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection Dec 16 15:50:00.696278880 (264h21m0.567361925s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696581516 (264h21m0.567664562s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 15:50:00.696601345 (264h21m0.56768439s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696644046 (264h21m0.567727091s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection Dec 16 15:50:00.696655878 (264h21m0.567738923s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696680695 (264h21m0.567763741s) [seg_pos={0 0}, upload=231735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:50:00.696775707 (264h21m0.567858752s) [upload=231735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:50:00.696781778 (264h21m0.567864823s) [upload=231735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:50:00.998597868 (264h21m0.869680913s) [upload=231735]: Done uploading segments... Dec 16 15:50:00.998625229 (264h21m0.869708264s) [upload=231735]: Upload finished. err: <nil> Duration: 5.784501124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.XX] duration: 5.001709186s trace: https://jaeger.dev.storj.io/trace/1290719905f75e2c FAIL (2025-12-16 15:51:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00173256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............................................X.] duration: 5.819840224s trace: https://jaeger.dev.storj.io/trace/2c8955a88808a9c7 FAIL (2025-12-16 15:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ) context deadline exceeded ======LOGS====== 1735]: Uploading segments... Dec 16 15:49:57.849123937 (264h20m57.720206992s) [upload=231735]: Got next segment. Inline: false Dec 16 15:49:57.849153704 (264h20m57.720236749s) [upload=231735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:49:57.849159384 (264h20m57.720242429s) [upload=231735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:49:58.100982669 (264h20m57.972065704s) [seg_pos={0 0}, upload=231735]: Begin upload segment... Dec 16 15:49:58.101029037 (264h20m57.972112072s) [seg_pos={0 0}, upload=231735]: Done begin upload segment. Dec 16 15:49:58.101096566 (264h20m57.972179601s) [noise=true , address=145.79.227.41:4085 , piece_id=4TQXRTBJUO725YUYWHF55IXTPMJZV4BNZN6AYN5QZRG3J7IR66OA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101118829 (264h20m57.972201864s) [noise=true , address=145.79.227.33:4869 , piece_id=3BO5NKJTJHTIAUM3TIVKJPUIPKQ34ICZ2MMO4FS6TB2FTJENVLOA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101574506 (264h20m57.972657541s) [noise=true , address=145.79.227.52:4446 , piece_id=IF6ASWX3KXUJBBTO66Q6VWWDM57UJETZTKBCGUW6QZW2JR7FLKTA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101890688 (264h20m57.972973733s) [noise=true , address=145.79.227.39:4014 , piece_id=6R2CLDIMLKPUTO5VLRJQLKHA6COBYH6EEQCEN6JDSIZOIWXHZTTA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102264310 (264h20m57.973347355s) [noise=true , address=145.79.227.3:4150 , piece_id=7GPTKGQYPELZMSLSFIZEEN7TNORVIU26Q27JBLF553YCYWYTMZYQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102564922 (264h20m57.973647967s) [noise=true , address=145.79.227.20:4437 , piece_id=4XGSBK3RWNOCSYOHUN4SRQ3JZ5LHC53RDH5MMF2RKKEF6Q5PBSUA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102856147 (264h20m57.973939192s) [noise=true , address=145.79.227.17:4970 , piece_id=LD3KKLNQEQKPQN36KNJA24JU3VFFMP6TJSXD2OY62KBSE3P4ZOKA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.103234788 (264h20m57.974317833s) [noise=true , address=145.79.227.46:4603 , piece_id=KNHAOIYBT4Y4AI7ZX37EWWO2PPWI7JDJUPXYFDCL3BPLNESCIDQQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.103630621 (264h20m57.974713656s) [noise=true , address=145.79.227.40:4050 , piece_id=KCTVDTVNAVOHKTPYJZVWMEVTR2A32FFUEGIBJTA77ZKDCIKJIO5A, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.105265925 (264h20m57.97634897s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.107092713 (264h20m57.978175748s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.108896789 (264h20m57.979979834s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.110314729 (264h20m57.981397774s) [noise=true , address=45.91.116.13:4156 , piece_id=RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.111559820 (264h20m57.982642855s) [noise=true , address=145.79.227.27:4690 , piece_id=5HUGLIYU3WRWBTBZAMHK3DCOJENPCDW3X63R7F47LLJMB6FB4RUA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.112108515 (264h20m57.98319155s) [noise=true , address=145.79.227.7:4293 , piece_id=UTII7KLLYGVDKKI6C2JFP5QL45KY2YYHK5ZAKWXLRE3YOTWQ32AA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.113391828 (264h20m57.984474863s) [noise=true , address=77.90.31.17:4773 , piece_id=FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:59.783147534 (264h20m59.654230569s) [upload=231735]: Next returned nil segment Dec 16 15:49:59.783160720 (264h20m59.654243765s) [upload=231735]: Waiting for error group managing segments... Dec 16 15:50:00.093340434 (264h20m59.96442348s) [noise=true , address=145.79.227.41:4085 , piece_id=4TQXRTBJUO725YUYWHF55IXTPMJZV4BNZN6AYN5QZRG3J7IR66OA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.099886569 (264h20m59.970969604s) [noise=true , address=145.79.227.52:4446 , piece_id=IF6ASWX3KXUJBBTO66Q6VWWDM57UJETZTKBCGUW6QZW2JR7FLKTA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.100217729 (264h20m59.971300775s) [noise=true , address=145.79.227.17:4970 , piece_id=LD3KKLNQEQKPQN36KNJA24JU3VFFMP6TJSXD2OY62KBSE3P4ZOKA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.102396048 (264h20m59.973479123s) [noise=true , address=145.79.227.27:4690 , piece_id=5HUGLIYU3WRWBTBZAMHK3DCOJENPCDW3X63R7F47LLJMB6FB4RUA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.107923433 (264h20m59.979006478s) [noise=true , address=145.79.227.39:4014 , piece_id=6R2CLDIMLKPUTO5VLRJQLKHA6COBYH6EEQCEN6JDSIZOIWXHZTTA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.113831112 (264h20m59.984914157s) [noise=true , address=145.79.227.20:4437 , piece_id=4XGSBK3RWNOCSYOHUN4SRQ3JZ5LHC53RDH5MMF2RKKEF6Q5PBSUA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.119556885 (264h20m59.99063993s) [noise=true , address=145.79.227.3:4150 , piece_id=7GPTKGQYPELZMSLSFIZEEN7TNORVIU26Q27JBLF553YCYWYTMZYQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.123421494 (264h20m59.994504529s) [noise=true , address=145.79.227.7:4293 , piece_id=UTII7KLLYGVDKKI6C2JFP5QL45KY2YYHK5ZAKWXLRE3YOTWQ32AA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.125340808 (264h20m59.996423853s) [noise=true , address=145.79.227.40:4050 , piece_id=KCTVDTVNAVOHKTPYJZVWMEVTR2A32FFUEGIBJTA77ZKDCIKJIO5A, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.162453147 (264h21m0.033536192s) [noise=true , address=145.79.227.46:4603 , piece_id=KNHAOIYBT4Y4AI7ZX37EWWO2PPWI7JDJUPXYFDCL3BPLNESCIDQQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.415415770 (264h21m0.286498825s) [noise=true , address=77.90.31.17:4773 , piece_id=FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.460990852 (264h21m0.332073887s) [noise=true , address=145.79.227.33:4869 , piece_id=3BO5NKJTJHTIAUM3TIVKJPUIPKQ34ICZ2MMO4FS6TB2FTJENVLOA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.695944363 (264h21m0.567027399s) [noise=true , address=45.91.116.13:4156 , piece_id=RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.695971876 (264h21m0.567054911s) [seg_pos={0 0}, upload=231735]: Segment reached optimal threshold of 13 pieces. Dec 16 15:50:00.696268691 (264h21m0.567351736s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection Dec 16 15:50:00.696278880 (264h21m0.567361925s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696581516 (264h21m0.567664562s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 15:50:00.696601345 (264h21m0.56768439s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696644046 (264h21m0.567727091s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection Dec 16 15:50:00.696655878 (264h21m0.567738923s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696680695 (264h21m0.567763741s) [seg_pos={0 0}, upload=231735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:50:00.696775707 (264h21m0.567858752s) [upload=231735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:50:00.696781778 (264h21m0.567864823s) [upload=231735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:50:00.998597868 (264h21m0.869680913s) [upload=231735]: Done uploading segments... Dec 16 15:50:00.998625229 (264h21m0.869708264s) [upload=231735]: Upload finished. err: <nil> Duration: 5.819868539s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.X] duration: 5.000732957s trace: https://jaeger.dev.storj.io/trace/189d5ba9b045f0fe FAIL (2025-12-16 15:50:28 UTC) orbiter warning: context deadline exceeded Duration: 5.000770849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............................................X] duration: 30.002441761s trace: https://jaeger.dev.storj.io/trace/2c8955a88808a9c7 FAIL (2025-12-16 15:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ) context deadline exceeded ======LOGS====== 1735]: Uploading segments... Dec 16 15:49:57.849123937 (264h20m57.720206992s) [upload=231735]: Got next segment. Inline: false Dec 16 15:49:57.849153704 (264h20m57.720236749s) [upload=231735]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:49:57.849159384 (264h20m57.720242429s) [upload=231735]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:49:58.100982669 (264h20m57.972065704s) [seg_pos={0 0}, upload=231735]: Begin upload segment... Dec 16 15:49:58.101029037 (264h20m57.972112072s) [seg_pos={0 0}, upload=231735]: Done begin upload segment. Dec 16 15:49:58.101096566 (264h20m57.972179601s) [noise=true , address=145.79.227.41:4085 , piece_id=4TQXRTBJUO725YUYWHF55IXTPMJZV4BNZN6AYN5QZRG3J7IR66OA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101118829 (264h20m57.972201864s) [noise=true , address=145.79.227.33:4869 , piece_id=3BO5NKJTJHTIAUM3TIVKJPUIPKQ34ICZ2MMO4FS6TB2FTJENVLOA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101574506 (264h20m57.972657541s) [noise=true , address=145.79.227.52:4446 , piece_id=IF6ASWX3KXUJBBTO66Q6VWWDM57UJETZTKBCGUW6QZW2JR7FLKTA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.101890688 (264h20m57.972973733s) [noise=true , address=145.79.227.39:4014 , piece_id=6R2CLDIMLKPUTO5VLRJQLKHA6COBYH6EEQCEN6JDSIZOIWXHZTTA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102264310 (264h20m57.973347355s) [noise=true , address=145.79.227.3:4150 , piece_id=7GPTKGQYPELZMSLSFIZEEN7TNORVIU26Q27JBLF553YCYWYTMZYQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102564922 (264h20m57.973647967s) [noise=true , address=145.79.227.20:4437 , piece_id=4XGSBK3RWNOCSYOHUN4SRQ3JZ5LHC53RDH5MMF2RKKEF6Q5PBSUA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.102856147 (264h20m57.973939192s) [noise=true , address=145.79.227.17:4970 , piece_id=LD3KKLNQEQKPQN36KNJA24JU3VFFMP6TJSXD2OY62KBSE3P4ZOKA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.103234788 (264h20m57.974317833s) [noise=true , address=145.79.227.46:4603 , piece_id=KNHAOIYBT4Y4AI7ZX37EWWO2PPWI7JDJUPXYFDCL3BPLNESCIDQQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.103630621 (264h20m57.974713656s) [noise=true , address=145.79.227.40:4050 , piece_id=KCTVDTVNAVOHKTPYJZVWMEVTR2A32FFUEGIBJTA77ZKDCIKJIO5A, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.105265925 (264h20m57.97634897s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.107092713 (264h20m57.978175748s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.108896789 (264h20m57.979979834s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.110314729 (264h20m57.981397774s) [noise=true , address=45.91.116.13:4156 , piece_id=RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.111559820 (264h20m57.982642855s) [noise=true , address=145.79.227.27:4690 , piece_id=5HUGLIYU3WRWBTBZAMHK3DCOJENPCDW3X63R7F47LLJMB6FB4RUA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.112108515 (264h20m57.98319155s) [noise=true , address=145.79.227.7:4293 , piece_id=UTII7KLLYGVDKKI6C2JFP5QL45KY2YYHK5ZAKWXLRE3YOTWQ32AA, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:58.113391828 (264h20m57.984474863s) [noise=true , address=77.90.31.17:4773 , piece_id=FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ, seg_pos={0 0}, upload=231735]: Uploading piece... Dec 16 15:49:59.783147534 (264h20m59.654230569s) [upload=231735]: Next returned nil segment Dec 16 15:49:59.783160720 (264h20m59.654243765s) [upload=231735]: Waiting for error group managing segments... Dec 16 15:50:00.093340434 (264h20m59.96442348s) [noise=true , address=145.79.227.41:4085 , piece_id=4TQXRTBJUO725YUYWHF55IXTPMJZV4BNZN6AYN5QZRG3J7IR66OA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.099886569 (264h20m59.970969604s) [noise=true , address=145.79.227.52:4446 , piece_id=IF6ASWX3KXUJBBTO66Q6VWWDM57UJETZTKBCGUW6QZW2JR7FLKTA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.100217729 (264h20m59.971300775s) [noise=true , address=145.79.227.17:4970 , piece_id=LD3KKLNQEQKPQN36KNJA24JU3VFFMP6TJSXD2OY62KBSE3P4ZOKA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.102396048 (264h20m59.973479123s) [noise=true , address=145.79.227.27:4690 , piece_id=5HUGLIYU3WRWBTBZAMHK3DCOJENPCDW3X63R7F47LLJMB6FB4RUA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.107923433 (264h20m59.979006478s) [noise=true , address=145.79.227.39:4014 , piece_id=6R2CLDIMLKPUTO5VLRJQLKHA6COBYH6EEQCEN6JDSIZOIWXHZTTA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.113831112 (264h20m59.984914157s) [noise=true , address=145.79.227.20:4437 , piece_id=4XGSBK3RWNOCSYOHUN4SRQ3JZ5LHC53RDH5MMF2RKKEF6Q5PBSUA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.119556885 (264h20m59.99063993s) [noise=true , address=145.79.227.3:4150 , piece_id=7GPTKGQYPELZMSLSFIZEEN7TNORVIU26Q27JBLF553YCYWYTMZYQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.123421494 (264h20m59.994504529s) [noise=true , address=145.79.227.7:4293 , piece_id=UTII7KLLYGVDKKI6C2JFP5QL45KY2YYHK5ZAKWXLRE3YOTWQ32AA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.125340808 (264h20m59.996423853s) [noise=true , address=145.79.227.40:4050 , piece_id=KCTVDTVNAVOHKTPYJZVWMEVTR2A32FFUEGIBJTA77ZKDCIKJIO5A, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.162453147 (264h21m0.033536192s) [noise=true , address=145.79.227.46:4603 , piece_id=KNHAOIYBT4Y4AI7ZX37EWWO2PPWI7JDJUPXYFDCL3BPLNESCIDQQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.415415770 (264h21m0.286498825s) [noise=true , address=77.90.31.17:4773 , piece_id=FD6IVQOBRNTCNMLZFGV3JZXQUM7FYBMHEWVD7GHR6DZODZIMNJMQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.460990852 (264h21m0.332073887s) [noise=true , address=145.79.227.33:4869 , piece_id=3BO5NKJTJHTIAUM3TIVKJPUIPKQ34ICZ2MMO4FS6TB2FTJENVLOA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.695944363 (264h21m0.567027399s) [noise=true , address=45.91.116.13:4156 , piece_id=RNKIJ6C4EOCHI6SJXQHMPRJ7PNBD2ZRDOVIORXRZNAIAJTFIPXOA, seg_pos={0 0}, upload=231735]: Done uploading piece. err: <nil> Dec 16 15:50:00.695971876 (264h21m0.567054911s) [seg_pos={0 0}, upload=231735]: Segment reached optimal threshold of 13 pieces. Dec 16 15:50:00.696268691 (264h21m0.567351736s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50234->77.90.31.11:4555: use of closed network connection Dec 16 15:50:00.696278880 (264h21m0.567361925s) [noise=true , address=77.90.31.11:4555 , piece_id=W2SRSCHR7CV2E6KWLFFS36WBI7VWL3EDPDYUET3SCQGMWW3YX6TQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696581516 (264h21m0.567664562s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 15:50:00.696601345 (264h21m0.56768439s) [noise=true , address=145.79.227.44:4194 , piece_id=E2ZM377KW7KOW5WLLNW4I7NNUV57ROVBFYZRSAIID4STAAXSKCRQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696644046 (264h21m0.567727091s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53884->77.90.31.10:4519: use of closed network connection Dec 16 15:50:00.696655878 (264h21m0.567738923s) [noise=true , address=77.90.31.10:4519 , piece_id=KXABF2PR3Y64CQOP2I5DI4CPC7VAJAJB3TZG5ZGNMUSFRUQ4CUTQ, seg_pos={0 0}, upload=231735]: Long tail cancellation due to optimal threshold Dec 16 15:50:00.696680695 (264h21m0.567763741s) [seg_pos={0 0}, upload=231735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:50:00.696775707 (264h21m0.567858752s) [upload=231735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:50:00.696781778 (264h21m0.567864823s) [upload=231735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:50:00.998597868 (264h21m0.869680913s) [upload=231735]: Done uploading segments... Dec 16 15:50:00.998625229 (264h21m0.869708264s) [upload=231735]: Upload finished. err: <nil> Duration: 30.002462881s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X.] duration: 3.57713517s trace: https://jaeger.dev.storj.io/trace/1856665e3756712c FAIL (2025-12-16 15:49:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577174944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............X.............................................] duration: 5.96027082s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.960300466s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.X] duration: 5.000596358s trace: https://jaeger.dev.storj.io/trace/1856665e3756712c FAIL (2025-12-16 15:48:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000619341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............X............................................] duration: 5.753395952s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.753428876s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [..XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX.] duration: 3.616898827s trace: https://jaeger.dev.storj.io/trace/79a17ace376e2f8e FAIL (2025-12-16 15:47:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.616943461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X.............X...........................................] duration: 5.995533425s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:47:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.995550247s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.XX] duration: 5.001865473s trace: https://jaeger.dev.storj.io/trace/79a17ace376e2f8e FAIL (2025-12-16 15:47:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001888949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X.............X..........................................] duration: 5.785900187s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:46:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.785926107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.X] duration: 5.001655634s trace: https://jaeger.dev.storj.io/trace/7611432b4c90fef6 FAIL (2025-12-16 15:46:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001680501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX.] duration: 4.383407086s trace: https://jaeger.dev.storj.io/trace/6ceb08181fa7e935 FAIL (2025-12-16 15:45:37 UTC) orbiter warning: context deadline exceeded Duration: 4.383458394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X.............X.........................................] duration: 5.876055259s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:45:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.876088352s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XXX] duration: 5.000794082s trace: https://jaeger.dev.storj.io/trace/6ceb08181fa7e935 FAIL (2025-12-16 15:45:21 UTC) orbiter warning: context deadline exceeded Duration: 5.000815733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.XX] duration: 5.001195037s trace: https://jaeger.dev.storj.io/trace/7be57f085a945158 FAIL (2025-12-16 15:45:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001219894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.X.............X........................................] duration: 6.007836513s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:44:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 6.00786653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.X] duration: 5.000984785s trace: https://jaeger.dev.storj.io/trace/5a125c38882e203e FAIL (2025-12-16 15:44:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001010225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.X.............X.......................................] duration: 5.704532482s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:43:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.704559033s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX.] duration: 3.782947901s trace: https://jaeger.dev.storj.io/trace/6896596e4f7dbc68 FAIL (2025-12-16 15:43:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.782972408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXXX] duration: 5.001625305s trace: https://jaeger.dev.storj.io/trace/6896596e4f7dbc68 FAIL (2025-12-16 15:43:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001652727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXXX] duration: 5.001086703s trace: https://jaeger.dev.storj.io/trace/586fa5ba8cf1ad2a FAIL (2025-12-16 15:42:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001108706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.X.............X......................................] duration: 5.787173734s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:42:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.787193362s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XXX] duration: 5.001701812s trace: https://jaeger.dev.storj.io/trace/25bb92d688bf06d2 FAIL (2025-12-16 15:42:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00172589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XXXXX.XX...XXXXXXX.X.XX.XXXXXXXX.XXXX..XXXXXXX..XXXXXXX.XX] duration: 5.001987486s trace: https://jaeger.dev.storj.io/trace/45a9ba4a2d59ae72 FAIL (2025-12-16 15:41:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002012634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.X.............X.....................................] duration: 5.745064379s trace: https://jaeger.dev.storj.io/trace/7ae4b670c22abaf9 FAIL (2025-12-16 15:41:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA) context deadline exceeded ======LOGS====== 6 15:13:15.963914692 (263h44m15.834997737s) [upload=231199]: Starting upload Dec 16 15:13:15.964035252 (263h44m15.835118297s) [upload=231199]: Done starting upload Dec 16 15:13:15.964227197 (263h44m15.835310232s) [upload=231199]: Uploading segments... Dec 16 15:13:15.964262564 (263h44m15.835345599s) [upload=231199]: Got next segment. Inline: false Dec 16 15:13:15.964293243 (263h44m15.835376288s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 15:13:15.964297861 (263h44m15.835380906s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 15:13:16.214698291 (263h44m16.085781356s) [seg_pos={0 0}, upload=231199]: Begin upload segment... Dec 16 15:13:16.214784535 (263h44m16.08586758s) [seg_pos={0 0}, upload=231199]: Done begin upload segment. Dec 16 15:13:16.214818820 (263h44m16.085901865s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.214887080 (263h44m16.085970125s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.215774039 (263h44m16.086857084s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.216588700 (263h44m16.087671735s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.217256872 (263h44m16.088339917s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.218190661 (263h44m16.089273716s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.219633188 (263h44m16.090716233s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.220041775 (263h44m16.09112482s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221456099 (263h44m16.092539144s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.221766650 (263h44m16.092849695s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.223220859 (263h44m16.094303905s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.226125971 (263h44m16.097209026s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.230831781 (263h44m16.101914816s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.235170163 (263h44m16.106253228s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.237409046 (263h44m16.108492081s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:16.242563130 (263h44m16.113646175s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Uploading piece... Dec 16 15:13:17.855030224 (263h44m17.726113259s) [upload=231199]: Next returned nil segment Dec 16 15:13:17.855048198 (263h44m17.726131243s) [upload=231199]: Waiting for error group managing segments... Dec 16 15:13:18.230429212 (263h44m18.101512277s) [noise=true , address=145.79.227.16:4909 , piece_id=H5P7C7TDPSDRGKHH5OFBLLUPHWSUAWHKNUNEHR22D6VMENWT67AA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.230558048 (263h44m18.101641093s) [noise=true , address=145.79.227.48:4301 , piece_id=UQYW3QC42NEHG2RO3YLV64Y326GM4HLHUE7NHW553KCHGUBWQRCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.242048057 (263h44m18.113131092s) [noise=true , address=145.79.227.22:4509 , piece_id=7NXRRPDMDBRAM3VFFX72HW37BBQ4GPMA6XGVCGJZM5CQNOCCUHVQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.245515670 (263h44m18.116598715s) [noise=true , address=145.79.227.42:4123 , piece_id=3ZFPMIFSAS3LILTNGMFJY6XLL2YMG525BOGUHUGMJNAMTE5ILCYA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.267110190 (263h44m18.138193235s) [noise=true , address=145.79.227.51:4411 , piece_id=F7FMENRWBJOWG3JH2XAWHZAKKWZ7QPKQ2WQQKVYJBB2HCEDCN2OA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.306201337 (263h44m18.177284392s) [noise=true , address=145.79.227.15:4876 , piece_id=XQQLSWVHJCQPZLLJZNKR6CI75A266XYJWH3F45JMRINBMNPSBS7A, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.320958507 (263h44m18.192041552s) [noise=true , address=145.79.227.36:4519 , piece_id=XU7DGJFP3TTFXI6S77YLW23HEYNF4RE5VLUMWBHO5574JZWLC5XA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.324379210 (263h44m18.195462255s) [noise=true , address=145.79.227.50:4373 , piece_id=MJBEH4MH6HAOHAEF74MLYN2BUOTKK5C5OG3ORBO3MVDB5ZJKGUCQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.328476673 (263h44m18.199559718s) [noise=true , address=145.79.227.38:4979 , piece_id=YKXM6VMFXZLD5K3TB66PPSNUOKTWLUKEJMIZTQGE6YOKAGCGP6WQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.332212917 (263h44m18.203295952s) [noise=true , address=145.79.227.35:4483 , piece_id=EASGXOJPBPEYV54FOZ2YR4K6FLBO3DSFU3KUE357FEXQKB2MZKOQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.500717992 (263h44m18.371801037s) [noise=true , address=77.90.31.14:4662 , piece_id=US6CTQ6QX2J7XZSDLDGNUYOY7GVNNZFWPZODXMIDAQFZCOF5BCJA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.522890913 (263h44m18.393973958s) [noise=true , address=145.79.227.2:4114 , piece_id=GI6RXRT3ZIEZALK3HH5AKXHPFC7DHSOPWN755ZUAP2QXDA74XCRQ, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.541988819 (263h44m18.413071864s) [noise=true , address=45.91.116.6:4975 , piece_id=IB4V6EHVCNKZUBJQCIUA5TOOVQTBV3L7SDR2TBLLOT2JBZD6INVA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: <nil> Dec 16 15:13:18.542016282 (263h44m18.413099327s) [seg_pos={0 0}, upload=231199]: Segment reached optimal threshold of 13 pieces. Dec 16 15:13:18.542287828 (263h44m18.413370863s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542296906 (263h44m18.413379951s) [noise=true , address=77.90.31.10:4518 , piece_id=QSCHOFG6WBBTXBG5DBWFPH5I4XOB32I6L7T5P74BXORMDZ5BM6JA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542337173 (263h44m18.413420218s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 16 15:13:18.542346119 (263h44m18.413429154s) [noise=true , address=45.91.116.10:4083 , piece_id=VBYHXRTW7MDUTIZO7JYXAVYLYQLPAG5YM3KB3UOTGYCQWKYPYHRA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542591026 (263h44m18.413674061s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->45.91.116.5:4297: use of closed network connection Dec 16 15:13:18.542598079 (263h44m18.413681124s) [noise=true , address=45.91.116.5:4297 , piece_id=GLB6UEJRGO5XQUQYLUP3NLEDAVUB2G4KEZIDQ3SNSUIRFL3KLHPA, seg_pos={0 0}, upload=231199]: Long tail cancellation due to optimal threshold Dec 16 15:13:18.542614931 (263h44m18.413697977s) [seg_pos={0 0}, upload=231199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 15:13:18.542698640 (263h44m18.413781676s) [upload=231199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 15:13:18.542704031 (263h44m18.413787066s) [upload=231199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 15:13:18.864544084 (263h44m18.735627129s) [upload=231199]: Done uploading segments... Dec 16 15:13:18.864581335 (263h44m18.73566437s) [upload=231199]: Upload finished. err: <nil> Duration: 5.745082825s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)