Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X........X...X..X....X...X..X.XX.XXX..X..X.........X.XX..X.] duration: 14.781089986s trace: https://jaeger.dev.storj.io/trace/547ccf9c20f3cf74 FAIL (2025-12-28 06:28:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: ISPGHQXNQQ6KEZODOOREY2UJAE7HOEHEROFSISULQY5BKN75UUDA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: CSYYI7IJBKZ5MAD374Z6Z2DXUPLKMVMD6O6QVX3HHAGZEDSU5O3A) context deadline exceeded ======LOGS====== ts... Dec 28 06:27:45.046472854 (542h58m44.917555899s) [upload=476160]: Got next segment. Inline: false Dec 28 06:27:45.047041616 (542h58m44.918124662s) [upload=476160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:27:45.047049451 (542h58m44.918132496s) [upload=476160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:27:45.291262429 (542h58m45.162345494s) [seg_pos={0 0}, upload=476160]: Begin upload segment... Dec 28 06:27:45.291367930 (542h58m45.162450985s) [seg_pos={0 0}, upload=476160]: Done begin upload segment. Dec 28 06:27:45.291398859 (542h58m45.162481904s) [noise=true , address=145.79.227.17:4971 , piece_id=5JCPIV4WU4ALQR2JMQD6O62RZNQRZEJZRST324QZ3ZDJWMEVLSYA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.291425179 (542h58m45.162508234s) [noise=true , address=77.90.31.16:4735 , piece_id=CSYYI7IJBKZ5MAD374Z6Z2DXUPLKMVMD6O6QVX3HHAGZEDSU5O3A, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.291535760 (542h58m45.162618805s) [noise=true , address=77.90.31.10:4518 , piece_id=ISPGHQXNQQ6KEZODOOREY2UJAE7HOEHEROFSISULQY5BKN75UUDA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.292028779 (542h58m45.163111814s) [noise=true , address=45.91.116.7:4010 , piece_id=UMBK3UXCFISMAN2UXHDVVZN3EVXE6TZYY4BRRNA7UTQJXCM2R2PQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.292503873 (542h58m45.163586918s) [noise=true , address=45.91.116.13:4153 , piece_id=6GT7WC2OTR5JZCU476A2M3EILRY7PRCOHLDTNIFJLLTG7L74WOUQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.292981042 (542h58m45.164064087s) [noise=true , address=145.79.227.50:4375 , piece_id=RUOW3HYIVDGBTXN7AX6HII2AI6IPVIBHMB6K2XXWIOPX2OAZDE4A, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.294354608 (542h58m45.165437653s) [noise=true , address=145.79.227.15:4876 , piece_id=SX4JEQPXYWZHRBOELTEUDVOXKPDBZGBLO5M7DQU2WFLBKZCNVIKA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.295409917 (542h58m45.166492952s) [noise=true , address=145.79.227.41:4087 , piece_id=SZ3NPNFDNYZ73D4T235G626HQ35YKYOSGKQYIOOQFRDWFJXRGUCA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.297352756 (542h58m45.168435802s) [noise=true , address=145.79.227.4:4185 , piece_id=NGKWYILHSMCHREOQFZPAIXKQNDKSMSFURX5Q2JYNDSXKLZVVYV3Q, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.299615225 (542h58m45.17069826s) [noise=true , address=145.79.227.24:4583 , piece_id=KIGLPQCUIXXZKNQD74FIGCHZYGOQAOBHVF7ETZMJ36EQOA4PTT6Q, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.301849769 (542h58m45.172932814s) [noise=true , address=145.79.227.45:4230 , piece_id=N4BJQCVNAOABBV3SS2ALKWGV2KXT2OZWVDBKIXJWBYMLIPTRGLYQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.303711304 (542h58m45.174794339s) [noise=true , address=77.90.31.14:4663 , piece_id=LGRJYFUT6YMJ6XT444SWV4GX57IR4ZJ56WLEMOGPPWB24HCKORKQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.305426120 (542h58m45.176509155s) [noise=true , address=145.79.227.37:4565 , piece_id=I2LPGI7MKT3JQZQRKE37FF2KDCBFIGCOSYMNUECUFELMO2WSISIQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.307506961 (542h58m45.178590006s) [noise=true , address=145.79.227.2:4113 , piece_id=4Y6UHT23HENUFO7TPRGZIOGAC2ODDQ3BQNEAHMEANYRPNH7GJBQA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.307922472 (542h58m45.179005507s) [noise=true , address=45.91.116.8:4048 , piece_id=MKB3BTRG22BH6NKW4BZU3NLYRBB67BED4LLC4TK6Z6I76LE5HB6Q, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.308400172 (542h58m45.179483207s) [noise=true , address=145.79.227.8:4331 , piece_id=VE4Z5XW3AUH65GYR5RSBZTRDZ6ZNSW3CZ65YGUAQWBNCQUPSADOQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:47.088691050 (542h58m46.959774085s) [upload=476160]: Next returned nil segment Dec 28 06:27:47.088736947 (542h58m46.959820013s) [upload=476160]: Waiting for error group managing segments... Dec 28 06:27:47.447256446 (542h58m47.318339481s) [noise=true , address=145.79.227.50:4375 , piece_id=RUOW3HYIVDGBTXN7AX6HII2AI6IPVIBHMB6K2XXWIOPX2OAZDE4A, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.461086000 (542h58m47.332169045s) [noise=true , address=145.79.227.41:4087 , piece_id=SZ3NPNFDNYZ73D4T235G626HQ35YKYOSGKQYIOOQFRDWFJXRGUCA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.464840259 (542h58m47.335923304s) [noise=true , address=145.79.227.17:4971 , piece_id=5JCPIV4WU4ALQR2JMQD6O62RZNQRZEJZRST324QZ3ZDJWMEVLSYA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.466336227 (542h58m47.337419263s) [noise=true , address=145.79.227.15:4876 , piece_id=SX4JEQPXYWZHRBOELTEUDVOXKPDBZGBLO5M7DQU2WFLBKZCNVIKA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.466601552 (542h58m47.337684587s) [noise=true , address=145.79.227.4:4185 , piece_id=NGKWYILHSMCHREOQFZPAIXKQNDKSMSFURX5Q2JYNDSXKLZVVYV3Q, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.467214790 (542h58m47.338297825s) [noise=true , address=145.79.227.24:4583 , piece_id=KIGLPQCUIXXZKNQD74FIGCHZYGOQAOBHVF7ETZMJ36EQOA4PTT6Q, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.468539943 (542h58m47.339622988s) [noise=true , address=145.79.227.8:4331 , piece_id=VE4Z5XW3AUH65GYR5RSBZTRDZ6ZNSW3CZ65YGUAQWBNCQUPSADOQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.489014181 (542h58m47.360097216s) [noise=true , address=145.79.227.2:4113 , piece_id=4Y6UHT23HENUFO7TPRGZIOGAC2ODDQ3BQNEAHMEANYRPNH7GJBQA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.489289465 (542h58m47.36037252s) [noise=true , address=145.79.227.45:4230 , piece_id=N4BJQCVNAOABBV3SS2ALKWGV2KXT2OZWVDBKIXJWBYMLIPTRGLYQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.520301057 (542h58m47.391384092s) [noise=true , address=45.91.116.7:4010 , piece_id=UMBK3UXCFISMAN2UXHDVVZN3EVXE6TZYY4BRRNA7UTQJXCM2R2PQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.521481124 (542h58m47.392564169s) [noise=true , address=45.91.116.13:4153 , piece_id=6GT7WC2OTR5JZCU476A2M3EILRY7PRCOHLDTNIFJLLTG7L74WOUQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.559908907 (542h58m47.430991952s) [noise=true , address=77.90.31.16:4735 , piece_id=CSYYI7IJBKZ5MAD374Z6Z2DXUPLKMVMD6O6QVX3HHAGZEDSU5O3A, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.581506453 (542h58m47.452589498s) [noise=true , address=77.90.31.10:4518 , piece_id=ISPGHQXNQQ6KEZODOOREY2UJAE7HOEHEROFSISULQY5BKN75UUDA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.581528234 (542h58m47.452611279s) [seg_pos={0 0}, upload=476160]: Segment reached optimal threshold of 13 pieces. Dec 28 06:27:47.581765416 (542h58m47.452848451s) [noise=true , address=45.91.116.8:4048 , piece_id=MKB3BTRG22BH6NKW4BZU3NLYRBB67BED4LLC4TK6Z6I76LE5HB6Q, seg_pos={0 0}, upload=476160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 06:27:47.581773741 (542h58m47.452856776s) [noise=true , address=45.91.116.8:4048 , piece_id=MKB3BTRG22BH6NKW4BZU3NLYRBB67BED4LLC4TK6Z6I76LE5HB6Q, seg_pos={0 0}, upload=476160]: Long tail cancellation due to optimal threshold Dec 28 06:27:47.582089904 (542h58m47.453172949s) [noise=true , address=77.90.31.14:4663 , piece_id=LGRJYFUT6YMJ6XT444SWV4GX57IR4ZJ56WLEMOGPPWB24HCKORKQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:55576->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55576->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55576->77.90.31.14:4663: use of closed network connection Dec 28 06:27:47.582098140 (542h58m47.453181175s) [noise=true , address=77.90.31.14:4663 , piece_id=LGRJYFUT6YMJ6XT444SWV4GX57IR4ZJ56WLEMOGPPWB24HCKORKQ, seg_pos={0 0}, upload=476160]: Long tail cancellation due to optimal threshold Dec 28 06:27:47.582212928 (542h58m47.453295973s) [noise=true , address=145.79.227.37:4565 , piece_id=I2LPGI7MKT3JQZQRKE37FF2KDCBFIGCOSYMNUECUFELMO2WSISIQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:51368->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:51368->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51368->145.79.227.37:4565: use of closed network connection Dec 28 06:27:47.582271880 (542h58m47.453354926s) [noise=true , address=145.79.227.37:4565 , piece_id=I2LPGI7MKT3JQZQRKE37FF2KDCBFIGCOSYMNUECUFELMO2WSISIQ, seg_pos={0 0}, upload=476160]: Long tail cancellation due to optimal threshold Dec 28 06:27:47.582363454 (542h58m47.453446489s) [seg_pos={0 0}, upload=476160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:27:47.582542526 (542h58m47.453625571s) [upload=476160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:27:47.582552475 (542h58m47.45363552s) [upload=476160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:27:47.875694393 (542h58m47.746777438s) [upload=476160]: Done uploading segments... Dec 28 06:27:47.875715904 (542h58m47.746798949s) [upload=476160]: Upload finished. err: <nil> Duration: 14.781120796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 40.311467527s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 40.311493278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X........X...X..X....X...X..X.XX.XXX..X..X.........X.XX..X] duration: 30.003731026s trace: https://jaeger.dev.storj.io/trace/547ccf9c20f3cf74 FAIL (2025-12-28 06:28:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: ISPGHQXNQQ6KEZODOOREY2UJAE7HOEHEROFSISULQY5BKN75UUDA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: CSYYI7IJBKZ5MAD374Z6Z2DXUPLKMVMD6O6QVX3HHAGZEDSU5O3A) context deadline exceeded ======LOGS====== ts... Dec 28 06:27:45.046472854 (542h58m44.917555899s) [upload=476160]: Got next segment. Inline: false Dec 28 06:27:45.047041616 (542h58m44.918124662s) [upload=476160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:27:45.047049451 (542h58m44.918132496s) [upload=476160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:27:45.291262429 (542h58m45.162345494s) [seg_pos={0 0}, upload=476160]: Begin upload segment... Dec 28 06:27:45.291367930 (542h58m45.162450985s) [seg_pos={0 0}, upload=476160]: Done begin upload segment. Dec 28 06:27:45.291398859 (542h58m45.162481904s) [noise=true , address=145.79.227.17:4971 , piece_id=5JCPIV4WU4ALQR2JMQD6O62RZNQRZEJZRST324QZ3ZDJWMEVLSYA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.291425179 (542h58m45.162508234s) [noise=true , address=77.90.31.16:4735 , piece_id=CSYYI7IJBKZ5MAD374Z6Z2DXUPLKMVMD6O6QVX3HHAGZEDSU5O3A, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.291535760 (542h58m45.162618805s) [noise=true , address=77.90.31.10:4518 , piece_id=ISPGHQXNQQ6KEZODOOREY2UJAE7HOEHEROFSISULQY5BKN75UUDA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.292028779 (542h58m45.163111814s) [noise=true , address=45.91.116.7:4010 , piece_id=UMBK3UXCFISMAN2UXHDVVZN3EVXE6TZYY4BRRNA7UTQJXCM2R2PQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.292503873 (542h58m45.163586918s) [noise=true , address=45.91.116.13:4153 , piece_id=6GT7WC2OTR5JZCU476A2M3EILRY7PRCOHLDTNIFJLLTG7L74WOUQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.292981042 (542h58m45.164064087s) [noise=true , address=145.79.227.50:4375 , piece_id=RUOW3HYIVDGBTXN7AX6HII2AI6IPVIBHMB6K2XXWIOPX2OAZDE4A, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.294354608 (542h58m45.165437653s) [noise=true , address=145.79.227.15:4876 , piece_id=SX4JEQPXYWZHRBOELTEUDVOXKPDBZGBLO5M7DQU2WFLBKZCNVIKA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.295409917 (542h58m45.166492952s) [noise=true , address=145.79.227.41:4087 , piece_id=SZ3NPNFDNYZ73D4T235G626HQ35YKYOSGKQYIOOQFRDWFJXRGUCA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.297352756 (542h58m45.168435802s) [noise=true , address=145.79.227.4:4185 , piece_id=NGKWYILHSMCHREOQFZPAIXKQNDKSMSFURX5Q2JYNDSXKLZVVYV3Q, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.299615225 (542h58m45.17069826s) [noise=true , address=145.79.227.24:4583 , piece_id=KIGLPQCUIXXZKNQD74FIGCHZYGOQAOBHVF7ETZMJ36EQOA4PTT6Q, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.301849769 (542h58m45.172932814s) [noise=true , address=145.79.227.45:4230 , piece_id=N4BJQCVNAOABBV3SS2ALKWGV2KXT2OZWVDBKIXJWBYMLIPTRGLYQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.303711304 (542h58m45.174794339s) [noise=true , address=77.90.31.14:4663 , piece_id=LGRJYFUT6YMJ6XT444SWV4GX57IR4ZJ56WLEMOGPPWB24HCKORKQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.305426120 (542h58m45.176509155s) [noise=true , address=145.79.227.37:4565 , piece_id=I2LPGI7MKT3JQZQRKE37FF2KDCBFIGCOSYMNUECUFELMO2WSISIQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.307506961 (542h58m45.178590006s) [noise=true , address=145.79.227.2:4113 , piece_id=4Y6UHT23HENUFO7TPRGZIOGAC2ODDQ3BQNEAHMEANYRPNH7GJBQA, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.307922472 (542h58m45.179005507s) [noise=true , address=45.91.116.8:4048 , piece_id=MKB3BTRG22BH6NKW4BZU3NLYRBB67BED4LLC4TK6Z6I76LE5HB6Q, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:45.308400172 (542h58m45.179483207s) [noise=true , address=145.79.227.8:4331 , piece_id=VE4Z5XW3AUH65GYR5RSBZTRDZ6ZNSW3CZ65YGUAQWBNCQUPSADOQ, seg_pos={0 0}, upload=476160]: Uploading piece... Dec 28 06:27:47.088691050 (542h58m46.959774085s) [upload=476160]: Next returned nil segment Dec 28 06:27:47.088736947 (542h58m46.959820013s) [upload=476160]: Waiting for error group managing segments... Dec 28 06:27:47.447256446 (542h58m47.318339481s) [noise=true , address=145.79.227.50:4375 , piece_id=RUOW3HYIVDGBTXN7AX6HII2AI6IPVIBHMB6K2XXWIOPX2OAZDE4A, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.461086000 (542h58m47.332169045s) [noise=true , address=145.79.227.41:4087 , piece_id=SZ3NPNFDNYZ73D4T235G626HQ35YKYOSGKQYIOOQFRDWFJXRGUCA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.464840259 (542h58m47.335923304s) [noise=true , address=145.79.227.17:4971 , piece_id=5JCPIV4WU4ALQR2JMQD6O62RZNQRZEJZRST324QZ3ZDJWMEVLSYA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.466336227 (542h58m47.337419263s) [noise=true , address=145.79.227.15:4876 , piece_id=SX4JEQPXYWZHRBOELTEUDVOXKPDBZGBLO5M7DQU2WFLBKZCNVIKA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.466601552 (542h58m47.337684587s) [noise=true , address=145.79.227.4:4185 , piece_id=NGKWYILHSMCHREOQFZPAIXKQNDKSMSFURX5Q2JYNDSXKLZVVYV3Q, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.467214790 (542h58m47.338297825s) [noise=true , address=145.79.227.24:4583 , piece_id=KIGLPQCUIXXZKNQD74FIGCHZYGOQAOBHVF7ETZMJ36EQOA4PTT6Q, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.468539943 (542h58m47.339622988s) [noise=true , address=145.79.227.8:4331 , piece_id=VE4Z5XW3AUH65GYR5RSBZTRDZ6ZNSW3CZ65YGUAQWBNCQUPSADOQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.489014181 (542h58m47.360097216s) [noise=true , address=145.79.227.2:4113 , piece_id=4Y6UHT23HENUFO7TPRGZIOGAC2ODDQ3BQNEAHMEANYRPNH7GJBQA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.489289465 (542h58m47.36037252s) [noise=true , address=145.79.227.45:4230 , piece_id=N4BJQCVNAOABBV3SS2ALKWGV2KXT2OZWVDBKIXJWBYMLIPTRGLYQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.520301057 (542h58m47.391384092s) [noise=true , address=45.91.116.7:4010 , piece_id=UMBK3UXCFISMAN2UXHDVVZN3EVXE6TZYY4BRRNA7UTQJXCM2R2PQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.521481124 (542h58m47.392564169s) [noise=true , address=45.91.116.13:4153 , piece_id=6GT7WC2OTR5JZCU476A2M3EILRY7PRCOHLDTNIFJLLTG7L74WOUQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.559908907 (542h58m47.430991952s) [noise=true , address=77.90.31.16:4735 , piece_id=CSYYI7IJBKZ5MAD374Z6Z2DXUPLKMVMD6O6QVX3HHAGZEDSU5O3A, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.581506453 (542h58m47.452589498s) [noise=true , address=77.90.31.10:4518 , piece_id=ISPGHQXNQQ6KEZODOOREY2UJAE7HOEHEROFSISULQY5BKN75UUDA, seg_pos={0 0}, upload=476160]: Done uploading piece. err: <nil> Dec 28 06:27:47.581528234 (542h58m47.452611279s) [seg_pos={0 0}, upload=476160]: Segment reached optimal threshold of 13 pieces. Dec 28 06:27:47.581765416 (542h58m47.452848451s) [noise=true , address=45.91.116.8:4048 , piece_id=MKB3BTRG22BH6NKW4BZU3NLYRBB67BED4LLC4TK6Z6I76LE5HB6Q, seg_pos={0 0}, upload=476160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 06:27:47.581773741 (542h58m47.452856776s) [noise=true , address=45.91.116.8:4048 , piece_id=MKB3BTRG22BH6NKW4BZU3NLYRBB67BED4LLC4TK6Z6I76LE5HB6Q, seg_pos={0 0}, upload=476160]: Long tail cancellation due to optimal threshold Dec 28 06:27:47.582089904 (542h58m47.453172949s) [noise=true , address=77.90.31.14:4663 , piece_id=LGRJYFUT6YMJ6XT444SWV4GX57IR4ZJ56WLEMOGPPWB24HCKORKQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:55576->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:55576->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55576->77.90.31.14:4663: use of closed network connection Dec 28 06:27:47.582098140 (542h58m47.453181175s) [noise=true , address=77.90.31.14:4663 , piece_id=LGRJYFUT6YMJ6XT444SWV4GX57IR4ZJ56WLEMOGPPWB24HCKORKQ, seg_pos={0 0}, upload=476160]: Long tail cancellation due to optimal threshold Dec 28 06:27:47.582212928 (542h58m47.453295973s) [noise=true , address=145.79.227.37:4565 , piece_id=I2LPGI7MKT3JQZQRKE37FF2KDCBFIGCOSYMNUECUFELMO2WSISIQ, seg_pos={0 0}, upload=476160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:51368->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:51368->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51368->145.79.227.37:4565: use of closed network connection Dec 28 06:27:47.582271880 (542h58m47.453354926s) [noise=true , address=145.79.227.37:4565 , piece_id=I2LPGI7MKT3JQZQRKE37FF2KDCBFIGCOSYMNUECUFELMO2WSISIQ, seg_pos={0 0}, upload=476160]: Long tail cancellation due to optimal threshold Dec 28 06:27:47.582363454 (542h58m47.453446489s) [seg_pos={0 0}, upload=476160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:27:47.582542526 (542h58m47.453625571s) [upload=476160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:27:47.582552475 (542h58m47.45363552s) [upload=476160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:27:47.875694393 (542h58m47.746777438s) [upload=476160]: Done uploading segments... Dec 28 06:27:47.875715904 (542h58m47.746798949s) [upload=476160]: Upload finished. err: <nil> Duration: 30.003772336s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X...........] duration: 3.424965227s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:27:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425028857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 47.27077899s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 47.270797215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X........X...X..X....X...X..X.XX.XXX..X..X.........X.XX..] duration: 13.225933653s trace: https://jaeger.dev.storj.io/trace/1009fa12c9ab1eb1 FAIL (2025-12-28 06:27:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 28 06:25:16.095293604 (542h56m15.966376659s) [upload=476121]: Got next segment. Inline: false Dec 28 06:25:16.095387243 (542h56m15.966470288s) [upload=476121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:25:16.095395639 (542h56m15.966478684s) [upload=476121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:25:16.337354884 (542h56m16.208437929s) [seg_pos={0 0}, upload=476121]: Begin upload segment... Dec 28 06:25:16.337403095 (542h56m16.208486131s) [seg_pos={0 0}, upload=476121]: Done begin upload segment. Dec 28 06:25:16.337446369 (542h56m16.208529404s) [noise=true , address=145.79.227.31:4797 , piece_id=AC3KXJTJGANAHJWC3VYZ2HQO6RBV3PRWA6RAOJ7VVSNZ4RI2Z5FA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.337483690 (542h56m16.208566725s) [noise=true , address=145.79.227.6:4257 , piece_id=UI7GI3MMXTA6XJ3RVE2HG2ZVNT5VOM4YP3MY2AEFV7ZCL66QO2LQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.338403280 (542h56m16.209486315s) [noise=true , address=145.79.227.13:4802 , piece_id=5O4IFDWYW6JUDSSXREEXQZW6D3ROBREATPHDVA5TXJMS3XABBATQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.338900146 (542h56m16.209983181s) [noise=true , address=145.79.227.37:4566 , piece_id=ATRYHIVGMBS2LXJQFOXLFEZYJAOVMRKMYW3CAXHZQJUS5PSJLMIQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.339348290 (542h56m16.210431325s) [noise=true , address=77.90.31.16:4736 , piece_id=7UMICKW4FWQZ667TG3W26VGCUFZGKGYN4JWXHRLMN7XU7IGZUHNA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.339728834 (542h56m16.210811869s) [noise=true , address=145.79.227.36:4519 , piece_id=Y7MRST3P7WBJ4H3NH24BBVNOAHKXKDTX6AHJETU7QCRN7KNHEMMA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340085253 (542h56m16.211168288s) [noise=true , address=45.91.116.4:4937 , piece_id=HAJEQWGB5EGL2ZP3AKYCV4GTHIIR4S7WWJOIOZ4BP327FFGRNLJQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340387809 (542h56m16.211470854s) [noise=true , address=145.79.227.4:4187 , piece_id=PS6BXGLD543MQY2LRHFLABRH5CCUBT6SJ7HNU2GAR3F24CC2NWXQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340708800 (542h56m16.211791845s) [noise=true , address=145.79.227.18:4006 , piece_id=CTJDWIFNLJACDVPMLGB7IW3LYYV2KHCEEOB6QTKX2BT5RO2VOUPQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.341093272 (542h56m16.212176317s) [noise=true , address=145.79.227.24:4583 , piece_id=UUFNAI44WH3SQVB2IONOGRX62J4AQBC3T7ZUULNXSIOMU5NNBNLA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.341965484 (542h56m16.213048519s) [noise=true , address=145.79.227.2:4113 , piece_id=OS4NV3BXWYEO7NC7PP3MNLVKN6IRWVB3554HRJSG7MZ4P5UJHDSQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.342479091 (542h56m16.213562126s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.342870097 (542h56m16.213953132s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343239650 (542h56m16.214322685s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGWKPVKWPKMOEXDZ6HVB45WVBV3K43IA4EHO6QGHI7VB42TRZA7A, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343575209 (542h56m16.214658254s) [noise=true , address=145.79.227.14:4839 , piece_id=XIWWDEZI357S6BBK75DLKE2CFG65IDLMKSJOGTNSSYGJ76NCFPAQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343924885 (542h56m16.21500793s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:17.974092077 (542h56m17.845175112s) [upload=476121]: Next returned nil segment Dec 28 06:25:17.974118087 (542h56m17.845201132s) [upload=476121]: Waiting for error group managing segments... Dec 28 06:25:18.352208342 (542h56m18.223291397s) [noise=true , address=145.79.227.36:4519 , piece_id=Y7MRST3P7WBJ4H3NH24BBVNOAHKXKDTX6AHJETU7QCRN7KNHEMMA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.352542949 (542h56m18.223625994s) [noise=true , address=145.79.227.13:4802 , piece_id=5O4IFDWYW6JUDSSXREEXQZW6D3ROBREATPHDVA5TXJMS3XABBATQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.356511756 (542h56m18.227594791s) [noise=true , address=145.79.227.24:4583 , piece_id=UUFNAI44WH3SQVB2IONOGRX62J4AQBC3T7ZUULNXSIOMU5NNBNLA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.357440986 (542h56m18.228524031s) [noise=true , address=145.79.227.2:4113 , piece_id=OS4NV3BXWYEO7NC7PP3MNLVKN6IRWVB3554HRJSG7MZ4P5UJHDSQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.365194130 (542h56m18.236277175s) [noise=true , address=145.79.227.18:4006 , piece_id=CTJDWIFNLJACDVPMLGB7IW3LYYV2KHCEEOB6QTKX2BT5RO2VOUPQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.372088648 (542h56m18.243171693s) [noise=true , address=145.79.227.31:4797 , piece_id=AC3KXJTJGANAHJWC3VYZ2HQO6RBV3PRWA6RAOJ7VVSNZ4RI2Z5FA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.372824048 (542h56m18.243907093s) [noise=true , address=145.79.227.6:4257 , piece_id=UI7GI3MMXTA6XJ3RVE2HG2ZVNT5VOM4YP3MY2AEFV7ZCL66QO2LQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.379162147 (542h56m18.250245382s) [noise=true , address=145.79.227.4:4187 , piece_id=PS6BXGLD543MQY2LRHFLABRH5CCUBT6SJ7HNU2GAR3F24CC2NWXQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.379883691 (542h56m18.250966746s) [noise=true , address=145.79.227.37:4566 , piece_id=ATRYHIVGMBS2LXJQFOXLFEZYJAOVMRKMYW3CAXHZQJUS5PSJLMIQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.380628269 (542h56m18.251711314s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGWKPVKWPKMOEXDZ6HVB45WVBV3K43IA4EHO6QGHI7VB42TRZA7A, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.381510288 (542h56m18.252593333s) [noise=true , address=145.79.227.14:4839 , piece_id=XIWWDEZI357S6BBK75DLKE2CFG65IDLMKSJOGTNSSYGJ76NCFPAQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.632130215 (542h56m18.50321325s) [noise=true , address=77.90.31.16:4736 , piece_id=7UMICKW4FWQZ667TG3W26VGCUFZGKGYN4JWXHRLMN7XU7IGZUHNA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.669532586 (542h56m18.540615631s) [noise=true , address=45.91.116.4:4937 , piece_id=HAJEQWGB5EGL2ZP3AKYCV4GTHIIR4S7WWJOIOZ4BP327FFGRNLJQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.669555069 (542h56m18.540638104s) [seg_pos={0 0}, upload=476121]: Segment reached optimal threshold of 13 pieces. Dec 28 06:25:18.669826116 (542h56m18.540909151s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection Dec 28 06:25:18.669836214 (542h56m18.54091925s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.669987612 (542h56m18.541070657s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection Dec 28 06:25:18.670005968 (542h56m18.541089003s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.670200028 (542h56m18.541283063s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 06:25:18.670208263 (542h56m18.541291298s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.670230886 (542h56m18.541313931s) [seg_pos={0 0}, upload=476121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:25:18.670455965 (542h56m18.54153902s) [upload=476121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:25:18.670463649 (542h56m18.541546684s) [upload=476121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:25:18.957775755 (542h56m18.828858801s) [upload=476121]: Done uploading segments... Dec 28 06:25:18.957842302 (542h56m18.828925347s) [upload=476121]: Upload finished. err: <nil> Duration: 13.225956917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X..........] duration: 4.304923742s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:26:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.304948338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 48.801443828s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:26:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 48.801463375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X........X...X..X....X...X..X.XX.XXX..X..X.........X.XX.] duration: 6.22710302s trace: https://jaeger.dev.storj.io/trace/1009fa12c9ab1eb1 FAIL (2025-12-28 06:26:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 28 06:25:16.095293604 (542h56m15.966376659s) [upload=476121]: Got next segment. Inline: false Dec 28 06:25:16.095387243 (542h56m15.966470288s) [upload=476121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:25:16.095395639 (542h56m15.966478684s) [upload=476121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:25:16.337354884 (542h56m16.208437929s) [seg_pos={0 0}, upload=476121]: Begin upload segment... Dec 28 06:25:16.337403095 (542h56m16.208486131s) [seg_pos={0 0}, upload=476121]: Done begin upload segment. Dec 28 06:25:16.337446369 (542h56m16.208529404s) [noise=true , address=145.79.227.31:4797 , piece_id=AC3KXJTJGANAHJWC3VYZ2HQO6RBV3PRWA6RAOJ7VVSNZ4RI2Z5FA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.337483690 (542h56m16.208566725s) [noise=true , address=145.79.227.6:4257 , piece_id=UI7GI3MMXTA6XJ3RVE2HG2ZVNT5VOM4YP3MY2AEFV7ZCL66QO2LQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.338403280 (542h56m16.209486315s) [noise=true , address=145.79.227.13:4802 , piece_id=5O4IFDWYW6JUDSSXREEXQZW6D3ROBREATPHDVA5TXJMS3XABBATQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.338900146 (542h56m16.209983181s) [noise=true , address=145.79.227.37:4566 , piece_id=ATRYHIVGMBS2LXJQFOXLFEZYJAOVMRKMYW3CAXHZQJUS5PSJLMIQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.339348290 (542h56m16.210431325s) [noise=true , address=77.90.31.16:4736 , piece_id=7UMICKW4FWQZ667TG3W26VGCUFZGKGYN4JWXHRLMN7XU7IGZUHNA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.339728834 (542h56m16.210811869s) [noise=true , address=145.79.227.36:4519 , piece_id=Y7MRST3P7WBJ4H3NH24BBVNOAHKXKDTX6AHJETU7QCRN7KNHEMMA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340085253 (542h56m16.211168288s) [noise=true , address=45.91.116.4:4937 , piece_id=HAJEQWGB5EGL2ZP3AKYCV4GTHIIR4S7WWJOIOZ4BP327FFGRNLJQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340387809 (542h56m16.211470854s) [noise=true , address=145.79.227.4:4187 , piece_id=PS6BXGLD543MQY2LRHFLABRH5CCUBT6SJ7HNU2GAR3F24CC2NWXQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340708800 (542h56m16.211791845s) [noise=true , address=145.79.227.18:4006 , piece_id=CTJDWIFNLJACDVPMLGB7IW3LYYV2KHCEEOB6QTKX2BT5RO2VOUPQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.341093272 (542h56m16.212176317s) [noise=true , address=145.79.227.24:4583 , piece_id=UUFNAI44WH3SQVB2IONOGRX62J4AQBC3T7ZUULNXSIOMU5NNBNLA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.341965484 (542h56m16.213048519s) [noise=true , address=145.79.227.2:4113 , piece_id=OS4NV3BXWYEO7NC7PP3MNLVKN6IRWVB3554HRJSG7MZ4P5UJHDSQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.342479091 (542h56m16.213562126s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.342870097 (542h56m16.213953132s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343239650 (542h56m16.214322685s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGWKPVKWPKMOEXDZ6HVB45WVBV3K43IA4EHO6QGHI7VB42TRZA7A, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343575209 (542h56m16.214658254s) [noise=true , address=145.79.227.14:4839 , piece_id=XIWWDEZI357S6BBK75DLKE2CFG65IDLMKSJOGTNSSYGJ76NCFPAQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343924885 (542h56m16.21500793s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:17.974092077 (542h56m17.845175112s) [upload=476121]: Next returned nil segment Dec 28 06:25:17.974118087 (542h56m17.845201132s) [upload=476121]: Waiting for error group managing segments... Dec 28 06:25:18.352208342 (542h56m18.223291397s) [noise=true , address=145.79.227.36:4519 , piece_id=Y7MRST3P7WBJ4H3NH24BBVNOAHKXKDTX6AHJETU7QCRN7KNHEMMA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.352542949 (542h56m18.223625994s) [noise=true , address=145.79.227.13:4802 , piece_id=5O4IFDWYW6JUDSSXREEXQZW6D3ROBREATPHDVA5TXJMS3XABBATQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.356511756 (542h56m18.227594791s) [noise=true , address=145.79.227.24:4583 , piece_id=UUFNAI44WH3SQVB2IONOGRX62J4AQBC3T7ZUULNXSIOMU5NNBNLA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.357440986 (542h56m18.228524031s) [noise=true , address=145.79.227.2:4113 , piece_id=OS4NV3BXWYEO7NC7PP3MNLVKN6IRWVB3554HRJSG7MZ4P5UJHDSQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.365194130 (542h56m18.236277175s) [noise=true , address=145.79.227.18:4006 , piece_id=CTJDWIFNLJACDVPMLGB7IW3LYYV2KHCEEOB6QTKX2BT5RO2VOUPQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.372088648 (542h56m18.243171693s) [noise=true , address=145.79.227.31:4797 , piece_id=AC3KXJTJGANAHJWC3VYZ2HQO6RBV3PRWA6RAOJ7VVSNZ4RI2Z5FA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.372824048 (542h56m18.243907093s) [noise=true , address=145.79.227.6:4257 , piece_id=UI7GI3MMXTA6XJ3RVE2HG2ZVNT5VOM4YP3MY2AEFV7ZCL66QO2LQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.379162147 (542h56m18.250245382s) [noise=true , address=145.79.227.4:4187 , piece_id=PS6BXGLD543MQY2LRHFLABRH5CCUBT6SJ7HNU2GAR3F24CC2NWXQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.379883691 (542h56m18.250966746s) [noise=true , address=145.79.227.37:4566 , piece_id=ATRYHIVGMBS2LXJQFOXLFEZYJAOVMRKMYW3CAXHZQJUS5PSJLMIQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.380628269 (542h56m18.251711314s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGWKPVKWPKMOEXDZ6HVB45WVBV3K43IA4EHO6QGHI7VB42TRZA7A, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.381510288 (542h56m18.252593333s) [noise=true , address=145.79.227.14:4839 , piece_id=XIWWDEZI357S6BBK75DLKE2CFG65IDLMKSJOGTNSSYGJ76NCFPAQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.632130215 (542h56m18.50321325s) [noise=true , address=77.90.31.16:4736 , piece_id=7UMICKW4FWQZ667TG3W26VGCUFZGKGYN4JWXHRLMN7XU7IGZUHNA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.669532586 (542h56m18.540615631s) [noise=true , address=45.91.116.4:4937 , piece_id=HAJEQWGB5EGL2ZP3AKYCV4GTHIIR4S7WWJOIOZ4BP327FFGRNLJQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.669555069 (542h56m18.540638104s) [seg_pos={0 0}, upload=476121]: Segment reached optimal threshold of 13 pieces. Dec 28 06:25:18.669826116 (542h56m18.540909151s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection Dec 28 06:25:18.669836214 (542h56m18.54091925s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.669987612 (542h56m18.541070657s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection Dec 28 06:25:18.670005968 (542h56m18.541089003s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.670200028 (542h56m18.541283063s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 06:25:18.670208263 (542h56m18.541291298s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.670230886 (542h56m18.541313931s) [seg_pos={0 0}, upload=476121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:25:18.670455965 (542h56m18.54153902s) [upload=476121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:25:18.670463649 (542h56m18.541546684s) [upload=476121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:25:18.957775755 (542h56m18.828858801s) [upload=476121]: Done uploading segments... Dec 28 06:25:18.957842302 (542h56m18.828925347s) [upload=476121]: Upload finished. err: <nil> Duration: 6.227127938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 39.83872591s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:25:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 39.838756668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X.........] duration: 5.000815231s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:25:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000839778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X........X...X..X....X...X..X.XX.XXX..X..X.........X.XX] duration: 30.002549805s trace: https://jaeger.dev.storj.io/trace/1009fa12c9ab1eb1 FAIL (2025-12-28 06:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 28 06:25:16.095293604 (542h56m15.966376659s) [upload=476121]: Got next segment. Inline: false Dec 28 06:25:16.095387243 (542h56m15.966470288s) [upload=476121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:25:16.095395639 (542h56m15.966478684s) [upload=476121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:25:16.337354884 (542h56m16.208437929s) [seg_pos={0 0}, upload=476121]: Begin upload segment... Dec 28 06:25:16.337403095 (542h56m16.208486131s) [seg_pos={0 0}, upload=476121]: Done begin upload segment. Dec 28 06:25:16.337446369 (542h56m16.208529404s) [noise=true , address=145.79.227.31:4797 , piece_id=AC3KXJTJGANAHJWC3VYZ2HQO6RBV3PRWA6RAOJ7VVSNZ4RI2Z5FA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.337483690 (542h56m16.208566725s) [noise=true , address=145.79.227.6:4257 , piece_id=UI7GI3MMXTA6XJ3RVE2HG2ZVNT5VOM4YP3MY2AEFV7ZCL66QO2LQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.338403280 (542h56m16.209486315s) [noise=true , address=145.79.227.13:4802 , piece_id=5O4IFDWYW6JUDSSXREEXQZW6D3ROBREATPHDVA5TXJMS3XABBATQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.338900146 (542h56m16.209983181s) [noise=true , address=145.79.227.37:4566 , piece_id=ATRYHIVGMBS2LXJQFOXLFEZYJAOVMRKMYW3CAXHZQJUS5PSJLMIQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.339348290 (542h56m16.210431325s) [noise=true , address=77.90.31.16:4736 , piece_id=7UMICKW4FWQZ667TG3W26VGCUFZGKGYN4JWXHRLMN7XU7IGZUHNA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.339728834 (542h56m16.210811869s) [noise=true , address=145.79.227.36:4519 , piece_id=Y7MRST3P7WBJ4H3NH24BBVNOAHKXKDTX6AHJETU7QCRN7KNHEMMA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340085253 (542h56m16.211168288s) [noise=true , address=45.91.116.4:4937 , piece_id=HAJEQWGB5EGL2ZP3AKYCV4GTHIIR4S7WWJOIOZ4BP327FFGRNLJQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340387809 (542h56m16.211470854s) [noise=true , address=145.79.227.4:4187 , piece_id=PS6BXGLD543MQY2LRHFLABRH5CCUBT6SJ7HNU2GAR3F24CC2NWXQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.340708800 (542h56m16.211791845s) [noise=true , address=145.79.227.18:4006 , piece_id=CTJDWIFNLJACDVPMLGB7IW3LYYV2KHCEEOB6QTKX2BT5RO2VOUPQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.341093272 (542h56m16.212176317s) [noise=true , address=145.79.227.24:4583 , piece_id=UUFNAI44WH3SQVB2IONOGRX62J4AQBC3T7ZUULNXSIOMU5NNBNLA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.341965484 (542h56m16.213048519s) [noise=true , address=145.79.227.2:4113 , piece_id=OS4NV3BXWYEO7NC7PP3MNLVKN6IRWVB3554HRJSG7MZ4P5UJHDSQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.342479091 (542h56m16.213562126s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.342870097 (542h56m16.213953132s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343239650 (542h56m16.214322685s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGWKPVKWPKMOEXDZ6HVB45WVBV3K43IA4EHO6QGHI7VB42TRZA7A, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343575209 (542h56m16.214658254s) [noise=true , address=145.79.227.14:4839 , piece_id=XIWWDEZI357S6BBK75DLKE2CFG65IDLMKSJOGTNSSYGJ76NCFPAQ, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:16.343924885 (542h56m16.21500793s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Uploading piece... Dec 28 06:25:17.974092077 (542h56m17.845175112s) [upload=476121]: Next returned nil segment Dec 28 06:25:17.974118087 (542h56m17.845201132s) [upload=476121]: Waiting for error group managing segments... Dec 28 06:25:18.352208342 (542h56m18.223291397s) [noise=true , address=145.79.227.36:4519 , piece_id=Y7MRST3P7WBJ4H3NH24BBVNOAHKXKDTX6AHJETU7QCRN7KNHEMMA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.352542949 (542h56m18.223625994s) [noise=true , address=145.79.227.13:4802 , piece_id=5O4IFDWYW6JUDSSXREEXQZW6D3ROBREATPHDVA5TXJMS3XABBATQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.356511756 (542h56m18.227594791s) [noise=true , address=145.79.227.24:4583 , piece_id=UUFNAI44WH3SQVB2IONOGRX62J4AQBC3T7ZUULNXSIOMU5NNBNLA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.357440986 (542h56m18.228524031s) [noise=true , address=145.79.227.2:4113 , piece_id=OS4NV3BXWYEO7NC7PP3MNLVKN6IRWVB3554HRJSG7MZ4P5UJHDSQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.365194130 (542h56m18.236277175s) [noise=true , address=145.79.227.18:4006 , piece_id=CTJDWIFNLJACDVPMLGB7IW3LYYV2KHCEEOB6QTKX2BT5RO2VOUPQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.372088648 (542h56m18.243171693s) [noise=true , address=145.79.227.31:4797 , piece_id=AC3KXJTJGANAHJWC3VYZ2HQO6RBV3PRWA6RAOJ7VVSNZ4RI2Z5FA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.372824048 (542h56m18.243907093s) [noise=true , address=145.79.227.6:4257 , piece_id=UI7GI3MMXTA6XJ3RVE2HG2ZVNT5VOM4YP3MY2AEFV7ZCL66QO2LQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.379162147 (542h56m18.250245382s) [noise=true , address=145.79.227.4:4187 , piece_id=PS6BXGLD543MQY2LRHFLABRH5CCUBT6SJ7HNU2GAR3F24CC2NWXQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.379883691 (542h56m18.250966746s) [noise=true , address=145.79.227.37:4566 , piece_id=ATRYHIVGMBS2LXJQFOXLFEZYJAOVMRKMYW3CAXHZQJUS5PSJLMIQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.380628269 (542h56m18.251711314s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGWKPVKWPKMOEXDZ6HVB45WVBV3K43IA4EHO6QGHI7VB42TRZA7A, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.381510288 (542h56m18.252593333s) [noise=true , address=145.79.227.14:4839 , piece_id=XIWWDEZI357S6BBK75DLKE2CFG65IDLMKSJOGTNSSYGJ76NCFPAQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.632130215 (542h56m18.50321325s) [noise=true , address=77.90.31.16:4736 , piece_id=7UMICKW4FWQZ667TG3W26VGCUFZGKGYN4JWXHRLMN7XU7IGZUHNA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.669532586 (542h56m18.540615631s) [noise=true , address=45.91.116.4:4937 , piece_id=HAJEQWGB5EGL2ZP3AKYCV4GTHIIR4S7WWJOIOZ4BP327FFGRNLJQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: <nil> Dec 28 06:25:18.669555069 (542h56m18.540638104s) [seg_pos={0 0}, upload=476121]: Segment reached optimal threshold of 13 pieces. Dec 28 06:25:18.669826116 (542h56m18.540909151s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33954->45.91.116.9:4262: use of closed network connection Dec 28 06:25:18.669836214 (542h56m18.54091925s) [noise=true , address=45.91.116.9:4262 , piece_id=K2TME5T6LKVP3QHMNLTKLXRFI3UAXE3OJQPLK7TZDBU73TU67XUA, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.669987612 (542h56m18.541070657s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->77.90.31.18:4805: use of closed network connection Dec 28 06:25:18.670005968 (542h56m18.541089003s) [noise=true , address=77.90.31.18:4805 , piece_id=2JX2SW4J2CHITJ3PWIZ7LE26HOBII67IFJSNTGWJSP6V4QO27TOQ, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.670200028 (542h56m18.541283063s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 06:25:18.670208263 (542h56m18.541291298s) [noise=true , address=77.90.31.14:4661 , piece_id=LCZXHQXQTE2JMXWOQZEMUG774J4URTDTHZKAL32TTWIR6KMIE52A, seg_pos={0 0}, upload=476121]: Long tail cancellation due to optimal threshold Dec 28 06:25:18.670230886 (542h56m18.541313931s) [seg_pos={0 0}, upload=476121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:25:18.670455965 (542h56m18.54153902s) [upload=476121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:25:18.670463649 (542h56m18.541546684s) [upload=476121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:25:18.957775755 (542h56m18.828858801s) [upload=476121]: Done uploading segments... Dec 28 06:25:18.957842302 (542h56m18.828925347s) [upload=476121]: Upload finished. err: <nil> Duration: 30.002566697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.....X........X...X..X....X...X..X.XX.XXX..X..X.........X.X] duration: 30.004030328s trace: https://jaeger.dev.storj.io/trace/4b214ecade8e6233 FAIL (2025-12-28 06:25:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: JEE4CUUELEOTI2UQ7RPCD5GRZZNVHDNS5KF55C75KWXFH2OEAOGA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MYPO7H3ZTRVTLKFHFRDRXFOKCMGSPJSPULD3PEAVOSS6FPZ37RHA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: I54JOYVE24JNOXDYCIKP6BKSWU3SUMKGGURC3UUQD3MJ5Y7QG2ZQ) context deadline exceeded ======LOGS====== c 28 06:24:45.876443008 (542h55m45.747526053s) [upload=476115]: Starting upload Dec 28 06:24:45.876497601 (542h55m45.747580646s) [upload=476115]: Done starting upload Dec 28 06:24:45.876717691 (542h55m45.747800726s) [upload=476115]: Uploading segments... Dec 28 06:24:45.876899156 (542h55m45.747982201s) [upload=476115]: Got next segment. Inline: false Dec 28 06:24:45.877028082 (542h55m45.748111127s) [upload=476115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:24:45.877035456 (542h55m45.748118491s) [upload=476115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:24:46.147625069 (542h55m46.018708104s) [seg_pos={0 0}, upload=476115]: Begin upload segment... Dec 28 06:24:46.147671808 (542h55m46.018754843s) [seg_pos={0 0}, upload=476115]: Done begin upload segment. Dec 28 06:24:46.147698198 (542h55m46.018781233s) [noise=true , address=145.79.227.39:4014 , piece_id=EIADWRYAKVH4SMS7ZKCEYALENHF5ZCDQFJTBABCL4WLWETC3GRZA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.147778861 (542h55m46.018861896s) [noise=true , address=77.90.31.15:4697 , piece_id=I54JOYVE24JNOXDYCIKP6BKSWU3SUMKGGURC3UUQD3MJ5Y7QG2ZQ, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.148309182 (542h55m46.019392217s) [noise=true , address=145.79.227.25:4619 , piece_id=KZRXNKKQUJKX77LLEDBRSWETGNTO4JGG237PNHULFT2SWZSFCWLQ, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.148477832 (542h55m46.019560867s) [noise=true , address=145.79.227.5:4221 , piece_id=RGCX4OQUV3PWM7VUWQCOVBBV4REMAPBZL5YNHBVPDXYGEKSTGIIA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.148799676 (542h55m46.019882721s) [noise=true , address=45.91.116.5:4297 , piece_id=VLK5OKPK3E7CMNO7TT2KLK4WTYZENVWMWZBPAL77A5LP5C3D25BQ, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.148978556 (542h55m46.020061591s) [noise=true , address=145.79.227.4:4185 , piece_id=6GWYDDYIGK2GD7DDNHD26E55FHUJDMQWRJ6KX3MW7232RWN4FZWQ, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.149275862 (542h55m46.020358897s) [noise=true , address=45.91.116.10:4084 , piece_id=JEE4CUUELEOTI2UQ7RPCD5GRZZNVHDNS5KF55C75KWXFH2OEAOGA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.149284719 (542h55m46.020367755s) [noise=true , address=45.91.116.7:4011 , piece_id=7NLPU7DDJMFZAXR33L3LNKQHXVMKFMUQ7GJD37RTJEB2NKON2OSA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.149568309 (542h55m46.020651344s) [noise=true , address=45.91.116.6:4973 , piece_id=MYPO7H3ZTRVTLKFHFRDRXFOKCMGSPJSPULD3PEAVOSS6FPZ37RHA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.149702525 (542h55m46.02078557s) [noise=true , address=145.79.227.48:4301 , piece_id=IOKYQHYBVXFFIFX7YB2AXFXVOJDXI5QFYHYS7JA64ELBBIWIVK5A, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.150006894 (542h55m46.021089929s) [noise=true , address=145.79.227.11:4731 , piece_id=GFV5GWLOA2KFNLNHCZ46LY2EBGG5T4372MIYQRIRPCMYAFGDEHPQ, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.150200022 (542h55m46.021283057s) [noise=true , address=45.91.116.11:4120 , piece_id=DSGWNI2KR66EZ3H723H7A333ICR2SL4WB5XEBP5MHB7BGKF5SYTA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.150408579 (542h55m46.021491624s) [noise=true , address=145.79.227.47:4266 , piece_id=KLO6GASJNVHQ776ME4NWMTKKA2OJA6SDVQVIZJKWUXXCRKRBXLVA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.150703541 (542h55m46.021786576s) [noise=true , address=145.79.227.52:4446 , piece_id=N4H332NU2NISLUAI5IGJL3B7MMNDTD5BVK3YWSNWGMUCHU6IVG4Q, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.150735802 (542h55m46.021818837s) [noise=true , address=45.91.116.4:4939 , piece_id=S43D7HN6FDKCN3JLJ35EQLVHUHXQL6QM2WIPCX2JYYGMAPFTMFIA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:46.151186861 (542h55m46.022269906s) [noise=true , address=77.90.31.21:4880 , piece_id=BQ3ETUBAHECSRU4T5MNFCKYH2VTIVIXBWV6A6ERWCJ3X25R55DRA, seg_pos={0 0}, upload=476115]: Uploading piece... Dec 28 06:24:47.871151205 (542h55m47.74223425s) [upload=476115]: Next returned nil segment Dec 28 06:24:47.871171544 (542h55m47.742254579s) [upload=476115]: Waiting for error group managing segments... Dec 28 06:24:48.161319490 (542h55m48.032402535s) [noise=true , address=145.79.227.39:4014 , piece_id=EIADWRYAKVH4SMS7ZKCEYALENHF5ZCDQFJTBABCL4WLWETC3GRZA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.162529504 (542h55m48.033612539s) [noise=true , address=145.79.227.11:4731 , piece_id=GFV5GWLOA2KFNLNHCZ46LY2EBGG5T4372MIYQRIRPCMYAFGDEHPQ, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.167997676 (542h55m48.039080711s) [noise=true , address=145.79.227.25:4619 , piece_id=KZRXNKKQUJKX77LLEDBRSWETGNTO4JGG237PNHULFT2SWZSFCWLQ, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.169526027 (542h55m48.040609062s) [noise=true , address=145.79.227.48:4301 , piece_id=IOKYQHYBVXFFIFX7YB2AXFXVOJDXI5QFYHYS7JA64ELBBIWIVK5A, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.174055031 (542h55m48.045138076s) [noise=true , address=145.79.227.5:4221 , piece_id=RGCX4OQUV3PWM7VUWQCOVBBV4REMAPBZL5YNHBVPDXYGEKSTGIIA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.174743802 (542h55m48.045826837s) [noise=true , address=145.79.227.52:4446 , piece_id=N4H332NU2NISLUAI5IGJL3B7MMNDTD5BVK3YWSNWGMUCHU6IVG4Q, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.190853769 (542h55m48.061936804s) [noise=true , address=145.79.227.4:4185 , piece_id=6GWYDDYIGK2GD7DDNHD26E55FHUJDMQWRJ6KX3MW7232RWN4FZWQ, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.192012535 (542h55m48.06309557s) [noise=true , address=145.79.227.47:4266 , piece_id=KLO6GASJNVHQ776ME4NWMTKKA2OJA6SDVQVIZJKWUXXCRKRBXLVA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.424711637 (542h55m48.295794672s) [noise=true , address=77.90.31.15:4697 , piece_id=I54JOYVE24JNOXDYCIKP6BKSWU3SUMKGGURC3UUQD3MJ5Y7QG2ZQ, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.476123435 (542h55m48.347206481s) [noise=true , address=45.91.116.6:4973 , piece_id=MYPO7H3ZTRVTLKFHFRDRXFOKCMGSPJSPULD3PEAVOSS6FPZ37RHA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.516605922 (542h55m48.387688967s) [noise=true , address=45.91.116.11:4120 , piece_id=DSGWNI2KR66EZ3H723H7A333ICR2SL4WB5XEBP5MHB7BGKF5SYTA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.671342169 (542h55m48.542425204s) [noise=true , address=45.91.116.10:4084 , piece_id=JEE4CUUELEOTI2UQ7RPCD5GRZZNVHDNS5KF55C75KWXFH2OEAOGA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.746414150 (542h55m48.617497195s) [noise=true , address=45.91.116.4:4939 , piece_id=S43D7HN6FDKCN3JLJ35EQLVHUHXQL6QM2WIPCX2JYYGMAPFTMFIA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: <nil> Dec 28 06:24:48.746437975 (542h55m48.61752102s) [seg_pos={0 0}, upload=476115]: Segment reached optimal threshold of 13 pieces. Dec 28 06:24:48.746653827 (542h55m48.617736872s) [noise=true , address=77.90.31.21:4880 , piece_id=BQ3ETUBAHECSRU4T5MNFCKYH2VTIVIXBWV6A6ERWCJ3X25R55DRA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 06:24:48.746664497 (542h55m48.617747552s) [noise=true , address=77.90.31.21:4880 , piece_id=BQ3ETUBAHECSRU4T5MNFCKYH2VTIVIXBWV6A6ERWCJ3X25R55DRA, seg_pos={0 0}, upload=476115]: Long tail cancellation due to optimal threshold Dec 28 06:24:48.746721976 (542h55m48.617805041s) [noise=true , address=45.91.116.7:4011 , piece_id=7NLPU7DDJMFZAXR33L3LNKQHXVMKFMUQ7GJD37RTJEB2NKON2OSA, seg_pos={0 0}, upload=476115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 06:24:48.746732837 (542h55m48.617815892s) [noise=true , address=45.91.116.7:4011 , piece_id=7NLPU7DDJMFZAXR33L3LNKQHXVMKFMUQ7GJD37RTJEB2NKON2OSA, seg_pos={0 0}, upload=476115]: Long tail cancellation due to optimal threshold Dec 28 06:24:48.746883834 (542h55m48.617966879s) [noise=true , address=45.91.116.5:4297 , piece_id=VLK5OKPK3E7CMNO7TT2KLK4WTYZENVWMWZBPAL77A5LP5C3D25BQ, seg_pos={0 0}, upload=476115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:34914->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:34914->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34914->45.91.116.5:4297: use of closed network connection Dec 28 06:24:48.746895016 (542h55m48.617978051s) [noise=true , address=45.91.116.5:4297 , piece_id=VLK5OKPK3E7CMNO7TT2KLK4WTYZENVWMWZBPAL77A5LP5C3D25BQ, seg_pos={0 0}, upload=476115]: Long tail cancellation due to optimal threshold Dec 28 06:24:48.746908942 (542h55m48.617991977s) [seg_pos={0 0}, upload=476115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:24:48.746970570 (542h55m48.618053605s) [upload=476115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:24:48.746974757 (542h55m48.618057792s) [upload=476115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:24:49.042064594 (542h55m48.91314763s) [upload=476115]: Done uploading segments... Dec 28 06:24:49.042084753 (542h55m48.913167798s) [upload=476115]: Upload finished. err: <nil> Duration: 30.004055074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X........] duration: 3.43755343s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:25:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437592904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 31.499965032s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 31.49997959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X.......] duration: 5.001186015s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:24:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001209229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.....X........X...X..X....X...X..X.XX.XXX..X..X.........X.] duration: 5.931619907s trace: https://jaeger.dev.storj.io/trace/29490ea7d121e5fd FAIL (2025-12-28 06:24:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: D2CU6LLUTNBUF2QL5RAX6L2ZL346HSMNY6TUVOCMV56JOL465OIQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: MAASAKRNLPXZAN3ENEU7ROSQLBFXICX4WFRV6EE63KL3RSBE4XZQ) context deadline exceeded ======LOGS====== Dec 28 06:23:13.718455846 (542h54m13.589538881s) [upload=476093]: Starting upload Dec 28 06:23:13.718533874 (542h54m13.589616919s) [upload=476093]: Done starting upload Dec 28 06:23:13.718657440 (542h54m13.589740485s) [upload=476093]: Uploading segments... Dec 28 06:23:13.718758362 (542h54m13.589841407s) [upload=476093]: Got next segment. Inline: false Dec 28 06:23:13.718802957 (542h54m13.589886002s) [upload=476093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:23:13.718809139 (542h54m13.589892184s) [upload=476093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:23:13.965673495 (542h54m13.83675654s) [seg_pos={0 0}, upload=476093]: Begin upload segment... Dec 28 06:23:13.965736275 (542h54m13.83681932s) [seg_pos={0 0}, upload=476093]: Done begin upload segment. Dec 28 06:23:13.965778756 (542h54m13.836861791s) [noise=true , address=45.91.116.7:4011 , piece_id=T6JD6IUUTJLRKB3LKGW5JDIWZKZMKOBM4MEJBHUAA76SQBNBRGKQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.965953879 (542h54m13.837036924s) [noise=true , address=45.91.116.4:4940 , piece_id=KB4Q64MWQVXONAUQSGRIKA6XPOEZU4L7OTC7TAFUYBVOHST3Q26Q, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.966641889 (542h54m13.837724924s) [noise=true , address=145.79.227.39:4014 , piece_id=KD4MUCPNCY7KVBS3ECIPPCVSIKCBVKOXBZMIAGLGUMQHKMY54RCQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.967059374 (542h54m13.838142419s) [noise=true , address=77.90.31.21:4877 , piece_id=2JRVN7SPIVEMCBGE5XD4VMA64RKBQDTZODJ4SZMIRNNFZMMWKK4A, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.967450970 (542h54m13.838534015s) [noise=true , address=145.79.227.14:4837 , piece_id=4E5HGBLVN7ZJG7APK3RYTLJQNIPA543HCTVFWNU6YWWNW5V3UXFA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.967846934 (542h54m13.838929979s) [noise=true , address=145.79.227.16:4910 , piece_id=POBKEYWSBFDKSDDOKA2SBIWMNQ7ZX4G7RFZF7C6AU2356QLIFJPQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.968281351 (542h54m13.839364396s) [noise=true , address=145.79.227.31:4799 , piece_id=JKDO5TILQ7CJYXOTH2TJBPVQOAUSXJ32AVY5WHLBKFWJRTT4NPDA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.968729955 (542h54m13.839812991s) [noise=true , address=77.90.31.11:4556 , piece_id=MAASAKRNLPXZAN3ENEU7ROSQLBFXICX4WFRV6EE63KL3RSBE4XZQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.969234276 (542h54m13.840317311s) [noise=true , address=145.79.227.2:4113 , piece_id=E4XJX56PQTLVTU6ZSLG64NHCEBQLYMCFDT7TUITC4FS574EJWIHA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.970669429 (542h54m13.841752474s) [noise=true , address=145.79.227.42:4122 , piece_id=B37CMNWCA7YZVCVN4RCWCCQSTS2NQQ3DAIZBNQLQDQSVT6WT3PRQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.971378809 (542h54m13.842461854s) [noise=true , address=145.79.227.47:4266 , piece_id=EQWBRVNOWGAMNT65IJJIXZNW4EMOACHLKIVYD2S2JMT3JELBHRUQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.973848923 (542h54m13.844931958s) [noise=true , address=145.79.227.33:4871 , piece_id=ZBKZ7YZX3KWQ5DU3QFHJ33TVF577DUMFESRW3QLCG6UZWYQFA6NA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.975199575 (542h54m13.84628262s) [noise=true , address=145.79.227.6:4259 , piece_id=7XRRFINM4D3ZXFUE6AEXQSCLI54OSSKPHZHILNJYFQA66L3RASPQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.976297996 (542h54m13.847381031s) [noise=true , address=45.91.116.10:4084 , piece_id=D2CU6LLUTNBUF2QL5RAX6L2ZL346HSMNY6TUVOCMV56JOL465OIQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.977415214 (542h54m13.848498259s) [noise=true , address=45.91.116.9:4263 , piece_id=AUCFPNFZFWXYO5J4GZJN4WXRXBXV3YL47BF2A3PJ6WKDMNGEFGRA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.978749014 (542h54m13.849832059s) [noise=true , address=145.79.227.38:4978 , piece_id=NB2NPJCHAD5JAYFUMEFFSFT4BHOO2QHCMJ2DJY75IGWBWQ5WVUXQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:15.690342207 (542h54m15.561425242s) [upload=476093]: Next returned nil segment Dec 28 06:23:15.690360742 (542h54m15.561443787s) [upload=476093]: Waiting for error group managing segments... Dec 28 06:23:15.968447892 (542h54m15.839530937s) [noise=true , address=145.79.227.31:4799 , piece_id=JKDO5TILQ7CJYXOTH2TJBPVQOAUSXJ32AVY5WHLBKFWJRTT4NPDA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.979589878 (542h54m15.850672913s) [noise=true , address=145.79.227.2:4113 , piece_id=E4XJX56PQTLVTU6ZSLG64NHCEBQLYMCFDT7TUITC4FS574EJWIHA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.980465807 (542h54m15.851548852s) [noise=true , address=145.79.227.16:4910 , piece_id=POBKEYWSBFDKSDDOKA2SBIWMNQ7ZX4G7RFZF7C6AU2356QLIFJPQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.986626969 (542h54m15.857710014s) [noise=true , address=145.79.227.14:4837 , piece_id=4E5HGBLVN7ZJG7APK3RYTLJQNIPA543HCTVFWNU6YWWNW5V3UXFA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.988054327 (542h54m15.859137372s) [noise=true , address=145.79.227.6:4259 , piece_id=7XRRFINM4D3ZXFUE6AEXQSCLI54OSSKPHZHILNJYFQA66L3RASPQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.990985759 (542h54m15.862068794s) [noise=true , address=145.79.227.38:4978 , piece_id=NB2NPJCHAD5JAYFUMEFFSFT4BHOO2QHCMJ2DJY75IGWBWQ5WVUXQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.001727002 (542h54m15.872810077s) [noise=true , address=145.79.227.39:4014 , piece_id=KD4MUCPNCY7KVBS3ECIPPCVSIKCBVKOXBZMIAGLGUMQHKMY54RCQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.002159515 (542h54m15.873242581s) [noise=true , address=145.79.227.47:4266 , piece_id=EQWBRVNOWGAMNT65IJJIXZNW4EMOACHLKIVYD2S2JMT3JELBHRUQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.002779296 (542h54m15.873862361s) [noise=true , address=145.79.227.42:4122 , piece_id=B37CMNWCA7YZVCVN4RCWCCQSTS2NQQ3DAIZBNQLQDQSVT6WT3PRQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.217326383 (542h54m16.088409428s) [noise=true , address=45.91.116.7:4011 , piece_id=T6JD6IUUTJLRKB3LKGW5JDIWZKZMKOBM4MEJBHUAA76SQBNBRGKQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.251856475 (542h54m16.12293952s) [noise=true , address=77.90.31.21:4877 , piece_id=2JRVN7SPIVEMCBGE5XD4VMA64RKBQDTZODJ4SZMIRNNFZMMWKK4A, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.516361192 (542h54m16.387444227s) [noise=true , address=45.91.116.10:4084 , piece_id=D2CU6LLUTNBUF2QL5RAX6L2ZL346HSMNY6TUVOCMV56JOL465OIQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.855499734 (542h54m16.726582779s) [noise=true , address=77.90.31.11:4556 , piece_id=MAASAKRNLPXZAN3ENEU7ROSQLBFXICX4WFRV6EE63KL3RSBE4XZQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.855525904 (542h54m16.726608939s) [seg_pos={0 0}, upload=476093]: Segment reached optimal threshold of 13 pieces. Dec 28 06:23:16.855757956 (542h54m16.726840991s) [noise=true , address=145.79.227.33:4871 , piece_id=ZBKZ7YZX3KWQ5DU3QFHJ33TVF577DUMFESRW3QLCG6UZWYQFA6NA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 28 06:23:16.855766693 (542h54m16.726849728s) [noise=true , address=145.79.227.33:4871 , piece_id=ZBKZ7YZX3KWQ5DU3QFHJ33TVF577DUMFESRW3QLCG6UZWYQFA6NA, seg_pos={0 0}, upload=476093]: Long tail cancellation due to optimal threshold Dec 28 06:23:16.855852056 (542h54m16.726935111s) [noise=true , address=45.91.116.9:4263 , piece_id=AUCFPNFZFWXYO5J4GZJN4WXRXBXV3YL47BF2A3PJ6WKDMNGEFGRA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 06:23:16.855862666 (542h54m16.726945711s) [noise=true , address=45.91.116.9:4263 , piece_id=AUCFPNFZFWXYO5J4GZJN4WXRXBXV3YL47BF2A3PJ6WKDMNGEFGRA, seg_pos={0 0}, upload=476093]: Long tail cancellation due to optimal threshold Dec 28 06:23:16.855899987 (542h54m16.726983022s) [noise=true , address=45.91.116.4:4940 , piece_id=KB4Q64MWQVXONAUQSGRIKA6XPOEZU4L7OTC7TAFUYBVOHST3Q26Q, seg_pos={0 0}, upload=476093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 06:23:16.855907310 (542h54m16.726990355s) [noise=true , address=45.91.116.4:4940 , piece_id=KB4Q64MWQVXONAUQSGRIKA6XPOEZU4L7OTC7TAFUYBVOHST3Q26Q, seg_pos={0 0}, upload=476093]: Long tail cancellation due to optimal threshold Dec 28 06:23:16.855924153 (542h54m16.727007188s) [seg_pos={0 0}, upload=476093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:23:16.855998815 (542h54m16.72708185s) [upload=476093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:23:16.856003844 (542h54m16.727086879s) [upload=476093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:23:17.146063012 (542h54m17.017146057s) [upload=476093]: Done uploading segments... Dec 28 06:23:17.146081115 (542h54m17.017164151s) [upload=476093]: Upload finished. err: <nil> Duration: 5.931645996s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X......] duration: 4.794751214s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:24:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.79477474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 31.210934816s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:24:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 31.210950558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.....X........X...X..X....X...X..X.XX.XXX..X..X.........X] duration: 30.002711304s trace: https://jaeger.dev.storj.io/trace/29490ea7d121e5fd FAIL (2025-12-28 06:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: D2CU6LLUTNBUF2QL5RAX6L2ZL346HSMNY6TUVOCMV56JOL465OIQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: MAASAKRNLPXZAN3ENEU7ROSQLBFXICX4WFRV6EE63KL3RSBE4XZQ) context deadline exceeded ======LOGS====== Dec 28 06:23:13.718455846 (542h54m13.589538881s) [upload=476093]: Starting upload Dec 28 06:23:13.718533874 (542h54m13.589616919s) [upload=476093]: Done starting upload Dec 28 06:23:13.718657440 (542h54m13.589740485s) [upload=476093]: Uploading segments... Dec 28 06:23:13.718758362 (542h54m13.589841407s) [upload=476093]: Got next segment. Inline: false Dec 28 06:23:13.718802957 (542h54m13.589886002s) [upload=476093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:23:13.718809139 (542h54m13.589892184s) [upload=476093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:23:13.965673495 (542h54m13.83675654s) [seg_pos={0 0}, upload=476093]: Begin upload segment... Dec 28 06:23:13.965736275 (542h54m13.83681932s) [seg_pos={0 0}, upload=476093]: Done begin upload segment. Dec 28 06:23:13.965778756 (542h54m13.836861791s) [noise=true , address=45.91.116.7:4011 , piece_id=T6JD6IUUTJLRKB3LKGW5JDIWZKZMKOBM4MEJBHUAA76SQBNBRGKQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.965953879 (542h54m13.837036924s) [noise=true , address=45.91.116.4:4940 , piece_id=KB4Q64MWQVXONAUQSGRIKA6XPOEZU4L7OTC7TAFUYBVOHST3Q26Q, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.966641889 (542h54m13.837724924s) [noise=true , address=145.79.227.39:4014 , piece_id=KD4MUCPNCY7KVBS3ECIPPCVSIKCBVKOXBZMIAGLGUMQHKMY54RCQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.967059374 (542h54m13.838142419s) [noise=true , address=77.90.31.21:4877 , piece_id=2JRVN7SPIVEMCBGE5XD4VMA64RKBQDTZODJ4SZMIRNNFZMMWKK4A, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.967450970 (542h54m13.838534015s) [noise=true , address=145.79.227.14:4837 , piece_id=4E5HGBLVN7ZJG7APK3RYTLJQNIPA543HCTVFWNU6YWWNW5V3UXFA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.967846934 (542h54m13.838929979s) [noise=true , address=145.79.227.16:4910 , piece_id=POBKEYWSBFDKSDDOKA2SBIWMNQ7ZX4G7RFZF7C6AU2356QLIFJPQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.968281351 (542h54m13.839364396s) [noise=true , address=145.79.227.31:4799 , piece_id=JKDO5TILQ7CJYXOTH2TJBPVQOAUSXJ32AVY5WHLBKFWJRTT4NPDA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.968729955 (542h54m13.839812991s) [noise=true , address=77.90.31.11:4556 , piece_id=MAASAKRNLPXZAN3ENEU7ROSQLBFXICX4WFRV6EE63KL3RSBE4XZQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.969234276 (542h54m13.840317311s) [noise=true , address=145.79.227.2:4113 , piece_id=E4XJX56PQTLVTU6ZSLG64NHCEBQLYMCFDT7TUITC4FS574EJWIHA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.970669429 (542h54m13.841752474s) [noise=true , address=145.79.227.42:4122 , piece_id=B37CMNWCA7YZVCVN4RCWCCQSTS2NQQ3DAIZBNQLQDQSVT6WT3PRQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.971378809 (542h54m13.842461854s) [noise=true , address=145.79.227.47:4266 , piece_id=EQWBRVNOWGAMNT65IJJIXZNW4EMOACHLKIVYD2S2JMT3JELBHRUQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.973848923 (542h54m13.844931958s) [noise=true , address=145.79.227.33:4871 , piece_id=ZBKZ7YZX3KWQ5DU3QFHJ33TVF577DUMFESRW3QLCG6UZWYQFA6NA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.975199575 (542h54m13.84628262s) [noise=true , address=145.79.227.6:4259 , piece_id=7XRRFINM4D3ZXFUE6AEXQSCLI54OSSKPHZHILNJYFQA66L3RASPQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.976297996 (542h54m13.847381031s) [noise=true , address=45.91.116.10:4084 , piece_id=D2CU6LLUTNBUF2QL5RAX6L2ZL346HSMNY6TUVOCMV56JOL465OIQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.977415214 (542h54m13.848498259s) [noise=true , address=45.91.116.9:4263 , piece_id=AUCFPNFZFWXYO5J4GZJN4WXRXBXV3YL47BF2A3PJ6WKDMNGEFGRA, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:13.978749014 (542h54m13.849832059s) [noise=true , address=145.79.227.38:4978 , piece_id=NB2NPJCHAD5JAYFUMEFFSFT4BHOO2QHCMJ2DJY75IGWBWQ5WVUXQ, seg_pos={0 0}, upload=476093]: Uploading piece... Dec 28 06:23:15.690342207 (542h54m15.561425242s) [upload=476093]: Next returned nil segment Dec 28 06:23:15.690360742 (542h54m15.561443787s) [upload=476093]: Waiting for error group managing segments... Dec 28 06:23:15.968447892 (542h54m15.839530937s) [noise=true , address=145.79.227.31:4799 , piece_id=JKDO5TILQ7CJYXOTH2TJBPVQOAUSXJ32AVY5WHLBKFWJRTT4NPDA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.979589878 (542h54m15.850672913s) [noise=true , address=145.79.227.2:4113 , piece_id=E4XJX56PQTLVTU6ZSLG64NHCEBQLYMCFDT7TUITC4FS574EJWIHA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.980465807 (542h54m15.851548852s) [noise=true , address=145.79.227.16:4910 , piece_id=POBKEYWSBFDKSDDOKA2SBIWMNQ7ZX4G7RFZF7C6AU2356QLIFJPQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.986626969 (542h54m15.857710014s) [noise=true , address=145.79.227.14:4837 , piece_id=4E5HGBLVN7ZJG7APK3RYTLJQNIPA543HCTVFWNU6YWWNW5V3UXFA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.988054327 (542h54m15.859137372s) [noise=true , address=145.79.227.6:4259 , piece_id=7XRRFINM4D3ZXFUE6AEXQSCLI54OSSKPHZHILNJYFQA66L3RASPQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:15.990985759 (542h54m15.862068794s) [noise=true , address=145.79.227.38:4978 , piece_id=NB2NPJCHAD5JAYFUMEFFSFT4BHOO2QHCMJ2DJY75IGWBWQ5WVUXQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.001727002 (542h54m15.872810077s) [noise=true , address=145.79.227.39:4014 , piece_id=KD4MUCPNCY7KVBS3ECIPPCVSIKCBVKOXBZMIAGLGUMQHKMY54RCQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.002159515 (542h54m15.873242581s) [noise=true , address=145.79.227.47:4266 , piece_id=EQWBRVNOWGAMNT65IJJIXZNW4EMOACHLKIVYD2S2JMT3JELBHRUQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.002779296 (542h54m15.873862361s) [noise=true , address=145.79.227.42:4122 , piece_id=B37CMNWCA7YZVCVN4RCWCCQSTS2NQQ3DAIZBNQLQDQSVT6WT3PRQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.217326383 (542h54m16.088409428s) [noise=true , address=45.91.116.7:4011 , piece_id=T6JD6IUUTJLRKB3LKGW5JDIWZKZMKOBM4MEJBHUAA76SQBNBRGKQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.251856475 (542h54m16.12293952s) [noise=true , address=77.90.31.21:4877 , piece_id=2JRVN7SPIVEMCBGE5XD4VMA64RKBQDTZODJ4SZMIRNNFZMMWKK4A, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.516361192 (542h54m16.387444227s) [noise=true , address=45.91.116.10:4084 , piece_id=D2CU6LLUTNBUF2QL5RAX6L2ZL346HSMNY6TUVOCMV56JOL465OIQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.855499734 (542h54m16.726582779s) [noise=true , address=77.90.31.11:4556 , piece_id=MAASAKRNLPXZAN3ENEU7ROSQLBFXICX4WFRV6EE63KL3RSBE4XZQ, seg_pos={0 0}, upload=476093]: Done uploading piece. err: <nil> Dec 28 06:23:16.855525904 (542h54m16.726608939s) [seg_pos={0 0}, upload=476093]: Segment reached optimal threshold of 13 pieces. Dec 28 06:23:16.855757956 (542h54m16.726840991s) [noise=true , address=145.79.227.33:4871 , piece_id=ZBKZ7YZX3KWQ5DU3QFHJ33TVF577DUMFESRW3QLCG6UZWYQFA6NA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 28 06:23:16.855766693 (542h54m16.726849728s) [noise=true , address=145.79.227.33:4871 , piece_id=ZBKZ7YZX3KWQ5DU3QFHJ33TVF577DUMFESRW3QLCG6UZWYQFA6NA, seg_pos={0 0}, upload=476093]: Long tail cancellation due to optimal threshold Dec 28 06:23:16.855852056 (542h54m16.726935111s) [noise=true , address=45.91.116.9:4263 , piece_id=AUCFPNFZFWXYO5J4GZJN4WXRXBXV3YL47BF2A3PJ6WKDMNGEFGRA, seg_pos={0 0}, upload=476093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 06:23:16.855862666 (542h54m16.726945711s) [noise=true , address=45.91.116.9:4263 , piece_id=AUCFPNFZFWXYO5J4GZJN4WXRXBXV3YL47BF2A3PJ6WKDMNGEFGRA, seg_pos={0 0}, upload=476093]: Long tail cancellation due to optimal threshold Dec 28 06:23:16.855899987 (542h54m16.726983022s) [noise=true , address=45.91.116.4:4940 , piece_id=KB4Q64MWQVXONAUQSGRIKA6XPOEZU4L7OTC7TAFUYBVOHST3Q26Q, seg_pos={0 0}, upload=476093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 06:23:16.855907310 (542h54m16.726990355s) [noise=true , address=45.91.116.4:4940 , piece_id=KB4Q64MWQVXONAUQSGRIKA6XPOEZU4L7OTC7TAFUYBVOHST3Q26Q, seg_pos={0 0}, upload=476093]: Long tail cancellation due to optimal threshold Dec 28 06:23:16.855924153 (542h54m16.727007188s) [seg_pos={0 0}, upload=476093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:23:16.855998815 (542h54m16.72708185s) [upload=476093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:23:16.856003844 (542h54m16.727086879s) [upload=476093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:23:17.146063012 (542h54m17.017146057s) [upload=476093]: Done uploading segments... Dec 28 06:23:17.146081115 (542h54m17.017164151s) [upload=476093]: Upload finished. err: <nil> Duration: 30.002795875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 54.96067516s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 54.960702801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X.....] duration: 4.76819132s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:23:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.768398465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X....] duration: 5.001070666s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:22:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0010933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X.....X........X...X..X....X...X..X.XX.XXX..X..X.........] duration: 6.013983052s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 6.014010344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 26.961410691s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:22:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 26.961438093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X...] duration: 5.000734054s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:21:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000759702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X........] duration: 6.05788624s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:21:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 6.057907589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X.......] duration: 9.927631123s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:21:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 9.927662672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X....................X......................................] duration: 39.916651206s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:21:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 39.916681134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X..] duration: 5.001247073s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:20:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001262673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X.] duration: 3.545464911s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:20:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545491623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X......] duration: 6.057833811s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 6.05799576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X....................X.....................................] duration: 25.390198081s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:19:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 25.390213923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X....................X....................................] duration: 31.472383573s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:19:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 31.472440943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X.....] duration: 6.496433949s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:19:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 6.496463054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X....] duration: 24.844356018s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:19:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 24.844400963s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....X] duration: 5.002570301s trace: https://jaeger.dev.storj.io/trace/35bc616deb6d7724 FAIL (2025-12-28 06:19:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002703185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X....................X...................................] duration: 25.76685619s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:18:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 25.766874415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...X....................X..................................] duration: 40.399899632s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:18:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 40.399934838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX.....] duration: 3.422811338s trace: https://jaeger.dev.storj.io/trace/5d1c26d3731eba91 FAIL (2025-12-28 06:17:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422836535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X...] duration: 5.978968936s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:17:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 5.97900169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...X....................X.................................] duration: 36.567802093s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:17:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 36.567816611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X....X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX....] duration: 5.001267508s trace: https://jaeger.dev.storj.io/trace/5d1c26d3731eba91 FAIL (2025-12-28 06:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001296215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X...X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X..] duration: 5.958560732s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 5.958590239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...X....................X................................] duration: 41.350489969s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:17:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 41.350508343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X....X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX...] duration: 5.000787541s trace: https://jaeger.dev.storj.io/trace/5d1c26d3731eba91 FAIL (2025-12-28 06:16:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00081289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X...X.X..X.....X........X...X..X....X...X..X.XX.XXX..X..X.] duration: 6.223963758s trace: https://jaeger.dev.storj.io/trace/7e07738199519ed5 FAIL (2025-12-28 06:16:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA) context deadline exceeded ======LOGS====== Dec 28 06:15:14.705182249 (542h46m14.576265294s) [upload=475975]: Starting upload Dec 28 06:15:14.705239288 (542h46m14.576322333s) [upload=475975]: Done starting upload Dec 28 06:15:14.705406265 (542h46m14.5764893s) [upload=475975]: Uploading segments... Dec 28 06:15:14.705549379 (542h46m14.576632424s) [upload=475975]: Got next segment. Inline: false Dec 28 06:15:14.705607328 (542h46m14.576690373s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:15:14.705617468 (542h46m14.576700554s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:15:14.982418132 (542h46m14.853501167s) [seg_pos={0 0}, upload=475975]: Begin upload segment... Dec 28 06:15:14.982562978 (542h46m14.853646023s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983497878 (542h46m14.854580933s) [seg_pos={0 0}, upload=475975]: Done begin upload segment. Dec 28 06:15:14.983578481 (542h46m14.854661526s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.983988312 (542h46m14.855071357s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984373555 (542h46m14.85545659s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.984738831 (542h46m14.855821866s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985031779 (542h46m14.856114814s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985353541 (542h46m14.856436576s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.985652701 (542h46m14.856735776s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.986161910 (542h46m14.857244955s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.987467707 (542h46m14.858550742s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988049004 (542h46m14.85913205s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.988808891 (542h46m14.859891936s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.990092014 (542h46m14.861175049s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.991401197 (542h46m14.862484232s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.992625508 (542h46m14.863708544s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:14.993721396 (542h46m14.864804431s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Uploading piece... Dec 28 06:15:16.622842740 (542h46m16.493925785s) [upload=475975]: Next returned nil segment Dec 28 06:15:16.622857117 (542h46m16.493940162s) [upload=475975]: Waiting for error group managing segments... Dec 28 06:15:17.010653074 (542h46m16.881736119s) [noise=true , address=145.79.227.13:4802 , piece_id=ONX6QLJ7SMEO3EJKQVHIQSZUDJ4F4RDSN3G5NLF6B6LF27UUUKYA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.011280770 (542h46m16.882363805s) [noise=true , address=145.79.227.36:4519 , piece_id=VEZZCR4OKHG5KVKCKYYCHCIZPQMFM2UOYEYOXUALTXDOKP5WLL7A, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.017063691 (542h46m16.888146736s) [noise=true , address=145.79.227.23:4547 , piece_id=JLQYADK77I77O22DPYAKYYP4QNRWQFTOTJWSON5ZXPRAF7RF6EFQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.018005043 (542h46m16.889088088s) [noise=true , address=145.79.227.7:4293 , piece_id=QWNJWUXX2YMLY6AVJJZ6HVNUEANYGZVTQQN33F65ZDFFFUCN3BQQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.024107013 (542h46m16.895190058s) [noise=true , address=145.79.227.12:4767 , piece_id=QWSLHFOU6KB6X42KMGPU5SAK43EALD3MNDV62LEMLX6FSPCQTQTA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.034049976 (542h46m16.905133021s) [noise=true , address=145.79.227.5:4223 , piece_id=FY23JUPVJENWETWJZAVX4VFQZSFT7WPQJVSDE7TXDIEJF5LPLQGQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.045137418 (542h46m16.916220464s) [noise=true , address=145.79.227.2:4113 , piece_id=AJQHVXGRUCZHY2NZIBXXKWLBT75ELIHVENI73BOT6IHLBV4H4UDA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.197087016 (542h46m17.068170051s) [noise=true , address=45.91.116.14:4192 , piece_id=ODSCEAYCLARN5LTC2IKIAVRQMFIOS7FELOKNLIJUBOOMYOVJZRXA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.228485082 (542h46m17.099568117s) [noise=true , address=45.91.116.10:4083 , piece_id=LAIXDPFAHKEL7L3CAFCDHAVHZW2DU675RF76BVGH5TTYRL5E5PUA, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.238219218 (542h46m17.109302263s) [noise=true , address=45.91.116.9:4263 , piece_id=ATMOVA5DN5UPHPYDV5AJGWBHCR5AZNXPPAF6TFIPPIAESS4LGS7Q, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.245900223 (542h46m17.116983258s) [noise=true , address=45.91.116.6:4973 , piece_id=234QSMGGHS6QTT44ZNGFAWQDE3MSIMNIQ7746VZ7CVLPBREAR5BQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.248136762 (542h46m17.119219797s) [noise=true , address=45.91.116.8:4048 , piece_id=5NF5CMQJFUB2HSF5UXKMTHG6M5WNAAJE7GETVUIFQGLYK7FWJ5SQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304872967 (542h46m17.175956013s) [noise=true , address=77.90.31.18:4805 , piece_id=PDSGU27JKFAKYYQIDZOHD7UF6QIT3W2DJRWQY7XKNM22VYEAV4DQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: <nil> Dec 28 06:15:17.304906972 (542h46m17.175990017s) [seg_pos={0 0}, upload=475975]: Segment reached optimal threshold of 13 pieces. Dec 28 06:15:17.305167718 (542h46m17.176250763s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305178288 (542h46m17.176261333s) [noise=true , address=77.90.31.21:4878 , piece_id=OKKRDXDL3F3L4PEBHDH7LF23RMECVVEVGBLXK3CFM7AGQWJ7Y4YQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305560927 (542h46m17.176643982s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305570185 (542h46m17.17665324s) [noise=true , address=45.91.116.13:4155 , piece_id=O2AX5SPZXHBCTD5O5WCZ2PN4RCPDP57V4474BDAN32FFJNETH2FQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305725631 (542h46m17.176808676s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 06:15:17.305733796 (542h46m17.176816841s) [noise=true , address=45.91.116.12:4228 , piece_id=IT7SU6PB2VNPAWUBXP3BZNKTOGKNF7QZDDZEKRAN6EPHPYLMARXQ, seg_pos={0 0}, upload=475975]: Long tail cancellation due to optimal threshold Dec 28 06:15:17.305758784 (542h46m17.176841829s) [seg_pos={0 0}, upload=475975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:15:17.305888441 (542h46m17.176971486s) [upload=475975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:15:17.305895745 (542h46m17.17697879s) [upload=475975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:15:17.604603473 (542h46m17.475686528s) [upload=475975]: Done uploading segments... Dec 28 06:15:17.604628200 (542h46m17.475711245s) [upload=475975]: Upload finished. err: <nil> Duration: 6.223996029s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X....X...XX.X..XXX....X..XX.X...X.X.XX..X...X..XX.X...XX..] duration: 5.000635741s trace: https://jaeger.dev.storj.io/trace/5d1c26d3731eba91 FAIL (2025-12-28 06:16:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000695474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...X....................X...............................] duration: 28.203882648s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:16:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 28.203899532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...X....................X..............................] duration: 36.887394903s trace: https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6 FAIL (2025-12-28 06:15:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece... Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments... Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil> Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces. Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments... Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil> Duration: 36.887422826s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)